Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/67—Focus control based on electronic image sensor signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/67—Focus control based on electronic image sensor signals
  • H04N23/675—Focus control based on electronic image sensor signals comprising setting of focusing regions

Definitions

• the invention relates to the field of digital imaging. Specifically, the present invention is directed towards a method and apparatus for automatic focusing in an image capture system using symmetric FIR filters.
• Video capture may be used for such applications as video conferencing, video editing, and distributed video training.
• Still image capture with a digital camera may be used for such applications as photo albums, photo editing, and compositing.
• Each pixel element is responsible for capturing one of three- color channels: red, green, or blue. Specifically, each pixel element is made sensitive to a certain color channel through the use of a color filter placed over the pixel element such that the light energy reaching the pixel element is due only to the light energy from a particular spectrum. Each pixel element generates a signal that corresponds to the amount of light energy to which it is exposed.
• a method for determining a focus value in an image including selecting an area of interest in the image and a color plane of interest. Then, filtering the color plane of interest within the area of interest to produce a filtered region; and, determining the mean absolute value of the filtered region.
• a system for implementing the method is also disclosed.
• Figure 1 is a diagram illustrating an image with an area of interest.
• FIG 2 is a flow diagram illustrating the determination of a focal value (V) from an image such as the image in Figure 1 in accordance with one mode of operation of the present invention.
• Figure 3 contains the impulse and frequency responses (both the magnitude and phase responses) of a filter configured in accordance with one embodiment of the present invention.
• Figure 4 is a flow diagram illustrating one mode of operation of an auto- focusing system configured in accordance with the present invention.
• Figure 5 contains the graphs of the focal values of images from four series of images as determined using the flow diagram of Figure 2.
• the present invention provides an automatic focusing method and apparatus developed for digital image capture systems such as digital imaging and video cameras (to be generally referred to herein as "camera”).
• digital image capture systems such as digital imaging and video cameras
• camera to be generally referred to herein as "camera”
• a sequence of images at different focal lengths is captured.
• each image is filtered by a symmetric finite impulse response (FIR) filter.
• a focus value which indicates the level of camera focus, is derived from the filtered image.
• An FIR filter is adopted by the algorithm as it may be implemented by a fixed function digital signal processing (FFDSP) hardware with smaller computation costs and also avoid the error accumulation problem normally seen in infinite impulse response (IIR) filters.
• FFTDSP fixed function digital signal processing
• IIR infinite impulse response
• using a symmetric filter can reduce the number of multiplications for filtering the image roughly by half, as compared to an asymmetric filter.
• the focal distance at which the captured image has the largest focus value is considered the optimal focal distance
• a processor and a memory is used to process the images to extract focus values for determining when the image capturing system is in focus.
• the processor may be a digital signal processor or an application specific integrated circuit (ASIC).
• the processor may also be a general purpose processor.
• the memory may be any storage device suitable for access by the processor.
• Figure 1 is a diagram illustrating an image 100 with a width X and a height Y.
• Image 100 is composed of a set of pixels, each overlaid with a color filter from a color filter array (CFA) 102.
• CFA 102 is in a Bayer pattern, with a repeated red (R), green (G), green (G), and blue (B) filter pattern.
• Figure 1 also includes an area of interest 104, which has a size of N pixels and M rows.
• FIG 2 is a flow diagram illustrating the determination of a focal value (V) from an image such as image 100 of Figure 1 in accordance with one mode of operation of the present invention.
• an area of interest such as area of interest 104 is selected from an image such as image 100.
• an area of interest is selected from the image in the calculation of the focal value for the image.
• multiple areas of interest may be chosen, and multiple focal values may be calculated. The following description is directed towards determining one focal value per image.
• a color plane of interest is chosen from the area of interest.
• what is chosen is the green color plane, as the green spectrum is better suited for determining luminance.
• another color plane may be chosen with the requirement that the color plane chosen contains most of the luminance information of the scene. For example, in a Y-
• the yellow plane which contains most of the luminance information, would be chosen as the color plane of interest.
• the pixel at location (0,0) of the cropped image region e.g., area of interest 104 is set to be a green pixel. Specifically, the top-left corner of the cropped image region is placed so that the top-left pixel is a green pixel.
• every two green pixel columns in the cropped image region are merged into a single complete green pixel column to generate a green plane
• G(i,j) is the value of the green pixel at location (i,j) of the cropped image region.
• the G' is reduced to a size of M by N/2 pixels.
• This merge operation is used for the specific CFA pattern (e.g., the Bayer pattern) used in this image capturing system, but may be modified for any CFA pattern as necessary.
• merged color plane G' is filtered using a low-pass filter to reduce inaccuracies caused by noise (e.g., artifact edges introduced by the use of the Bayer pattern).
• the interpolated green plane ⁇ « remains the same size of M by N/2 pixels.
• operation then continues with block 208.
• the system divides the interpolated green plane G a into three sub-regions, G ⁇ , G 2 and G 3 f with equal sizes.
• the interpolated green plane G a is divided into three sub-regions of size M x N/6 pixels.
• the interpolated green plane G a may be divided into three sub-regions of size M/3 x N/2 pixels.
• the interpolated green plane G a may be divided into multiple regions of any size for other embodiments.
• each sub region r G 2 and G 3 are filtered by a p- tap FIR filter:
• the filter attempts to extract relevant edge information useful for determining whether the camera is in focus.
• the p-tap FIR filter used to filter the sub regions G > , G ⁇ and G 3 of interpolated green plane G . is a 20-tap symmetric FIR filter.
• a symmetric FIR filter is used in the algorithm because it can be implemented in current existing hardware with smaller computation costs and the number of multiplications required for filtering the image may be reduced roughly by half when compared to a non-symmetric FIR filter.
• the FIR filter does not suffer from error accumulation problems normally encountered by IIR filters.
• Table 1 contains one possible set of filter coefficients of the FIR filter that may be used to produce desired results.
• Table 1 The coefficients of the symmetric FIR filter.
• Figure 3 shows the impulse and frequency responses (both the magnitude and phase responses) of the filter designed.
• the filter has a magnitude response similar to a band-pass filter. After each sub-region has been filtered, operation then continues with block 212.
• a mean absolute value is computed for each filtered sub- region G ⁇ k :
• V(r) the focus value of current image r
• V(r) max .
• Figure 4 is a flow diagram illustrating one mode of operation of an auto- focusing system configured in accordance with the present invention.
• the scheme is presented with a series of images captured at different focal distances.
• the scheme locates a focal distance at which the image would be focused before a complete series of images at different focal distances are captured.
• the images to be processed by this scheme are captured as the scheme searches for the focused picture.
• the scheme performs a quick search to find a focal length that produces a relatively focused image (e.g., by examining the focal values to find the highest focal value available) and overshoots that focal length to where the focal value is smaller by a certain percentage than the previous focal value.
• the scheme performs a detailed search to determine a focal length that produces a more in-focus image.
• This scheme works under the assumption that the computed focus value is a unimodal function of the focal distance, which is the case of most of the scenes encountered.
• J f opt / J max where / is a focal distance; V(f) is the focal value that is computed for an image taken at/; V m ⁇ X is the largest focal value that has been found; _ ⁇ . « and mm are longest and shortest focal distances, respectively, capable by the image capture system; and P ⁇ is the optimal focal distance located by this scheme.
• a focal value, V(f), is determined for an image captured at the focal distance /.
• V(f) is determined following the description for Figure 2. As noted above, if the scheme is not presented with a series of captured images, then the system captures the image immediately during block 104, before the focal value is determined. After the focal value is computed in block 104, it is compared, in block 106, with the maximum focal value that has been found. If the focal value is greater than largest previously calculated focal value ( V( J ) > v mR , ) / operation continues with block 108. Otherwise, operation continues with block 110.
• the coarse focal distance searching step is a step size for changing the focal distance during the search for the coarse focal distance.
• the coarse focal distance searching step i is found by: where SN1 and SFN1 control the total number of image evaluations. SFN1 depends on the current F-Number setting as well as the current focal distance /. SFN1 may be tabularized or formulated with simple equations based on prior experiments.
• SN1 is a number that controls the number of searches within the coarse search.
• the step size may be larger as the image capturing system is in focus over a wider range of focal distances.
• the step size should be smaller as the image capturing system is in focus over a narrow range of focal distances.
• the scheme uses a predetermined percentage to evaluate whether the optimum focal value has been passed by the algorithm and the system is now evaluating images that are more and more out of focus (e.g., the focal value is becoming smaller and smaller). In other embodiments, the percentage may be variable based on the current focal distance or other parameters.
• the focal distance that provides a focussed image is contained in the range from the maximum focal distance supported by the image capture system, f max , to the current focal distance,/.
• the coarse search started from the maximum focal distance and examined the focal values of images captured at decreasing focal distances until a preset condition was met, it is logical that the focal distance of interest is contained in the range.
• the current focal distance is subtracted with the coarse focal distance searching step in block 110, and one of the conditions that would cause the coarse search to end is if the current focal distance is not greater than or equal to ⁇ , there is the possibility that the current focal distance is a value that is lower tha f mi ⁇ , which is not a valid value.
• a focal value, V(f), is determined for an image captured at the focal distance / V(f) is determined following the description for Figure 2.
• the focal value is computed in block 116, it is compared, in block 118, with the maximum focal value that has been found. If the focal value is greater than largest previously calculated focal value ( ) > ⁇ « ' ), operation continues with block 120. Otherwise, operation continues with block 122.
• the fine focal distance searching step is a step size for changing the focal distance during the search for the fine focal distance.
• the focal distance searching step tsi is found by:
• step size' is set much smaller than the step size ⁇ i to have provide higher fine tune capability.
• the step size may be larger as the image capturing system is in focus over a wider range of focal distances.
• the step size should be smaller as the image capturing system is in focus over a narrow range of focal distances.
• the scheme uses a predetermined percentage to evaluate whether the optimum focal value has been passed by the algorithm and the system is now evaluating images that are more and more out of focus (e.g., the focal value is becoming smaller and smaller). In other embodiments, the percentage may be variable based on the current focal distance or other parameters.
• the optimal focal distance, J pl , is output and the operation ends for the series of images (or, in the case where the system is capturing images as needed for evaluation of the focal value, the operation ends until a new focal distance is needed).
• Figure 5 illustrates the focus values for the images of four sequences (normalized to the maximum focus value within each test sequence).
• the images marked with circles (o) are well focused images indicated.
• the computed focus values may be user to identify well focused images for the four sequences.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Studio Devices (AREA)
• Automatic Focus Adjustment (AREA)
• Image Processing (AREA)
• Color Television Image Signal Generators (AREA)
• Focusing (AREA)
• Color Image Communication Systems (AREA)

Priority Applications (6)

Applications Claiming Priority (2)

Publications (1)

Family

ID=23511795

Family Applications (1)

Country Status (10)

Families Citing this family (31)

Citations (5)

Family Cites Families (9)

• 1999
  • 1999-08-25 US US09/383,117 patent/US6373481B1/en not_active Expired - Lifetime
• 2000
  • 2000-08-11 JP JP2001519029A patent/JP2003507769A/ja active Pending
  • 2000-08-11 GB GB0203899A patent/GB2370111B/en not_active Expired - Fee Related
  • 2000-08-11 WO PCT/US2000/022128 patent/WO2001015434A1/en not_active Ceased
  • 2000-08-11 KR KR10-2002-7002348A patent/KR100449835B1/ko not_active Expired - Fee Related
  • 2000-08-11 AU AU65388/00A patent/AU771694B2/en not_active Ceased
  • 2000-08-11 IL IL14824000A patent/IL148240A0/xx active IP Right Grant
  • 2000-08-11 RU RU2002107312/09A patent/RU2222113C2/ru not_active IP Right Cessation
  • 2000-08-11 DE DE10084921T patent/DE10084921B4/de not_active Expired - Fee Related
  • 2000-09-19 TW TW089117216A patent/TW490976B/zh not_active IP Right Cessation
• 2002
  • 2002-02-19 IL IL148240A patent/IL148240A/en unknown

Patent Citations (5)

Also Published As

Similar Documents

Legal Events