Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/76—Circuitry for compensating brightness variation in the scene by influencing the image signals
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/90—Dynamic range modification of images or parts thereof
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/20—Analysis of motion
  • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
  • G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/74—Circuitry for compensating brightness variation in the scene by influencing the scene brightness using illuminating means
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10141—Special mode during image acquisition
  • G06T2207/10152—Varying illumination
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30168—Image quality inspection
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30241—Trajectory

Definitions

• the present invention relates to a method for minimising error in ambient light corrected (ALC) images, and an image capturing apparatus for the same.
• Images captured by a camera or image capturing device of an object are often illuminated by ambient light as well as a separate source of illumination.
• the ambient light In order to be able to accurately compare images of the same object, or determine colours of an object, the ambient light must be corrected for to produce an ambient light corrected image.
• images are often generated by comparing pixels of captured images, which introduces errors when there is any movement of the object between frames.
• a method for minimising error in ambient light corrected image due to motion comprising: capturing a plurality of primary images with controlled intensity of illumination varying overtime; estimating an error value due to motion for each of a plurality of sub-sets of the plurality of primary images; and performing ambient light correction on an error minimised sub-set of primary images having the lowest estimated error value to generate an ambient light corrected (ALC) image with a minimised error; wherein estimating the error value for each sub-set of the plurality of primary images comprises: identifying a pair of primary images in the sub-set including a first primary image and a second primary image, wherein the first primary image is captured temporally before the second primary image; determining a displacement vector matrix between the pair of primary images, including: dividing the first primary image into a plurality of first grid elements and dividing the second primary image into a plurality of second grid elements, matching first grid elements to second grid elements which correspond to the same position on the object, and determining a displacement vector
• Estimating the error value for each sub-set of primary images may comprise identifying a plurality of pairs of primary images in the sub-set; determining the displacement vector matrix for each pair of primary images; and estimating the error value for the sub-set of the plurality of primary images based on the displacement vector matrix for each pair of primary images in the sub-set.
• Estimating the error value for the sub-set of the plurality of primary images may comprise selecting the highest absolute displacement vector from the maximum displacement vector for each pair of primary images, and estimating the error value based on the highest absolute displacement vector in the sub-set of primary images.
• Determining the displacement vector between each matched first grid element and second grid element may comprise performing a displacement estimation based on phase correlation.
• Each sub-set of primary images may comprise a plurality of temporally adjacent primary images. Each sub-set of primary images may be offset from another sub-set of primary images by one temporally adjacent primary image.
• Estimating the error value due to motion for each of a plurality of sub-sets may be performed in real time on temporally consecutive sub-sets of primary images.
• the method may comprise performing ambient light correction on the respective sub-set of primary images to generate an ALC image with a minimised error.
• a non-transitory computer readable storage medium comprising computer-readable instructions that, when executed by a processor, causes the performance of a method in accordance with the first aspect.
• a computer program that, when read by a computer, causes performance of a method in accordance with the first aspect.
• an image capturing apparatus comprising: at least one processor; at least one memory comprising computer-readable instructions; the at least one processor being configured to read the computer readable instructions and cause performance of a method in accordance with the first aspect.
• Fig. 1 schematically shows an image capturing apparatus for minimising error in ambient light corrected images
• Fig. 2 is a flow chart showing steps of a first example method for minimising error in ambient light corrected images
• Fig. 3 schematically shows steps of the first example method
• Fig. 4 is a flow chart showing steps of a second example method of estimating an error value for a sub-set of primary images
• FIGs. 5A-5C schematically show steps of the second example method.
• Fig. 6 is a flow chart showing steps of a third example method for minimising error in ambient light corrected images.
• Fig. 1 shows an image capturing apparatus 10 comprising an image capturing device 12, a light 14, a processor 16 and a memory 18 comprising computer readable instructions.
• the image capturing device 12 is connected to the processor 16 and is configured to capture images of an object 20, which in this example is a human, which can be read and processed by the processor 16.
• the light 14 is configured to illuminate the object 20 and is also connected to the processor 16.
• the light 14 is configured to emit variable intensity of illumination which is controlled by the processor 16.
• the light 14 may be controller by a separate controller.
• the memory 18 comprises computer readable instructions and the processor 16 is configured to read the computer readable instructions to perform a method for minimising error in ambient light corrected (ALC) images.
• ALC ambient light corrected
• Fig. 2 is a flow chart showing steps of a first example method 100 of minimising error in AFC images.
• Fig. 3 shows the steps of the method in terms of the outputs of captured and processed images.
• the first example method 100 comprises capturing a plurality of primary images 30 of the object 20 over time, where the object 20 is illuminated by the light 14, and where the illumination is controlled to vary over time.
• the passage of time is represented by the arrow 34.
• Each primary image 30 comprises a plurality of pixels 50 which each have a pixel index corresponding to the location of the pixel 50 in the image, and a pixel value defining the intensity of the light captured at that pixel 50.
• the method 100 comprises identifying a plurality of sub-sets of primary images 30.
• Each sub-set of primary images 30 comprises a predetermined number of temporally adjacent primary images 30 on which ambient light correction will be performed to generate an ambient light corrected (AFC) image 42.
• AFC ambient light corrected
• an AFC image is generated from two primary images 30 and so each sub-set of primary images 30 comprises two primary images 30.
• Each sub-set of primary images 30 is offset from a preceding sub-set of primary images 30 by one temporally adjacent primary image 30. In other examples, there may be an offset of more than one primary image between sub-sets of primary images.
• the method 100 comprises estimating an error value due to motion of each of a plurality of sub-sets of primary images 30.
• Block 106 of the method 100 comprises block 108, block 110, block 112, block 114, and block 116. Since the steps in block 106 are applied to each sub-set of primary images 30, the steps will be described below with respect to a single sub-set of primary images 30.
• the method 100 comprises identifying pairs of primary images 30 within the sub-set of primary images 30.
• the sub-set of primary images 30 comprises two primary images 30, and so there is only one pair of primary images 30.
• Each pair of primary images 30 is defined as a first primary image 30a, and a second primary image 30b, where the first primary image 30a is captured temporally before the second primary image 30b.
• the sub-set of primary images comprises more than two primary images, then within the sub-set of primary images, there will be more than one pair of primary images.
• the method 100 comprises dividing the first primary image 30a into a plurality of first grid elements and dividing the second primary image 30b into a plurality of second grid elements.
• Each grid element may comprise a one pixel 50 or a plurality of pixels 50.
• the method 100 comprises matching first grid elements which may each represent a position on the object 20, to second grid elements which correspond to the same position on the object 20.
• Matched first grid elements and second grid elements may comprise pixels 50 with different a pixel index due to movement of the object 20 between capturing the first primary image 30a and the second primary image 30b.
• the method 100 comprises determining a displacement vector between each matched first grid element and second grid element, describing the movement of the object 20 in terms of the movement from the first primary image 30a to the second primary image 30b, and determining a displacement vector matrix 36 describing the displacement vector of each matched grid element.
• the displacement vector matrix 36 is a list of vectors which may be recorded in an array corresponding to the grid elements, or may simply be a list of vectors which does not retain correspondence to the grid elements.
• determining a displacement vector for matched first grid elements and second grid elements may comprise performing a displacement estimation based on phase correlation. Using the phase information of received light at each pixel or grid element to determine a displacement vector, and ignoring the intensity of the light (i.e. the amplitude of the light), means that displacement estimation is more accurate, particularly when there is variation in light.
• the method 100 comprises estimating the error value 38 for the sub-set of primary images 30 based on the displacement vector matrix 36 between the pair of primary images 30. If there is more than one pair of primary images in the sub-set of primary images, then the error value for the sub-set of primary images is based on the displacement vector matrices of each of the pairs of primary images within the sub-set.
• estimating the error value 38 for the sub-set of primary images 30 comprises selecting the maximum displacement vector in the displacement vector matrix 36 for the pair of primary images 30 and estimating the error value based on the maximum displacement vector. If there are more than one pair of primary images 30 in the sub-set of primary images 30, estimating the error value 38 may comprise selecting the maximum displacement vector in the displacement vector matrices 36 of all of the pairs of primary images 30 within the sub-set of primary images 30, and estimating the error value 38 based on the maximum displacement vectors. In other examples, estimating the error value may comprise aggregating the displacement vectors by summing them, or determining a mean absolute displacement vector, for example.
• the output from block 106 is a plurality of error values 38, each corresponding to an estimated error due to motion for a respective sub-set of primary images 30.
• the method 100 comprises selecting an error minimised sub-set of primary images 40 having the lowest error value E x , and performing ambient light correction on the error minimised sub-set of primary images 40 to generate an ALC image 42 with a minimised error.
• a first primary image may be captured with the light switched on to illuminate the object, and a second primary image may be captured with the light switched off such that only ambient light illuminates the object.
• the intensity of the second primary image at that pixel can be subtracted to generate an ALC pixel, and combining all of the ALC pixels generates the ALC image.
• Fig. 4 is a flow chart showing steps of a second example method 200 of estimating the error value 38 for a sub-set of primary images 30 having more than two primary images 30.
• the second example method 200 may replace block 106 of the first example method 100.
• Figs. 5A, 5B and 5C schematically shows steps of the second example method 200.
• the second example method 200 produces the same basic outputs at various steps as the first example method 100, and therefore the description below will refer to reference numerals in Fig. 3.
• the second example method 200 comprises identifying a plurality of pairs of primary images 30 in the sub-set. For example, if a sub-set contains three primary images, the sub-set will comprise three independent pairs of primary images including (i) a first primary image and a second primary image, (ii) the second primary image and a third primary image, and (iii) the first primary image and the third primary image. In the example shown in Figs. 5A-5C, the sub-set comprises six primary images 30 which comprises 15 pairs of primary images 30.
• the method 200 comprises determining a displacement vector matrix 36 for each pair of temporally adjacent primary images 30 in a similar manner to blocks 110-114 of the first example method 100.
• Fig. 5A shows a first primary image 30a divided into a plurality of grid elements 46, and a displacement vector 48 is illustrated by an arrow for each grid element 46, where the beginning of the arrow is the location of the grid element on the first primary image 30a of the pair and the end of the arrow is the location of the matching grid element on the second primary image 30b of the pair.
• the plurality of displacement vectors 48 for the plurality of grid elements 46 forms the displacement vector matrix 36 between the first primary image 30a and the second primary image 30b of a pair of temporally adjacent primary images 30.
• the method 200 comprises determining a displacement vector matrix 36 for temporally non-adjacent pairs of primary images 30 by adding displacement vectors 48 for each grid element 46 of temporally adjacent pairs of primary images 30, to map a motion trajectory 52 between the pair of temporally non-adjacent primary images 30.
• Fig. 5B shows the motion trajectory 52 for a grid element 46 between a first primary image and a sixth primary image by aggregating the displacement vectors 48 between temporally adjacent pairs of primary images 30.
• the displacement vector matrix for temporally non- adjacent primary images may be calculated in a similar manner as the displacement vector matrix for temporally adjacent primary images. Determining the displacement vector matrix for temporally non- adjacent pairs of images by mapping a motion trajectory requires less processing power than matching grid elements and calculating the displacement vectors for matched grid elements between the temporally non-adjacent pair of primary images.
• Fig. 5C shows displacement vectors 48 between temporally non-adjacent pairs of primary images superimposed on the motion trajectory 52.
• the method 200 comprises estimating the error value 38 for the sub-set of primary images 30 based on the displacement vector matrix 36 between each of the pairs of primary images 30 within the sub-set.
• estimating the error value for the sub-set of the plurality of primary images 30 comprises determining a maximum displacement vector for each pair of primary images 30 within the sub-set of primary images 30, and selecting the highest absolute displacement vector from the maximum displacement vectors.
• the error value 38 may be based on the highest absolute displacement vector.
• This method 200 can be repeated for every sub-set of primary images 30 to estimate an error value, and the method can continue to block 118 of the first example method 100 to select an error minimised sub-set of primary images 40 having the lowest error value E x , and perform ambient light correction on the error minimised sub-set of primary images 40 to generate an ALC image 42 with a minimised error.
• Fig. 6 shows a third example method 300 for minimising error in ALC images due to motion.
• the method 300 comprises capturing a plurality of primary images 30 in a similar manner to block 102 of the first example method 100.
• the method comprises identifying a sub-set of primary images 30 comprising a predetermined number of temporally adjacent primary images 30 on which ambient light correction could be performed to generate an ambient light corrected (ALC) image 42, in a similar manner to block 104 of the first example method 100.
• ALC ambient light corrected
• the method 300 comprises estimating an error value 38 due to motion of the sub-set of primary images 30. This may be in a manner similar to blocks 202-208 of the second example method 200, or block 106 of the first example method 100.
• block 308 it is determined whether the error value 38 of the sub-set of primary images 30 is the lowest error value found so far. If it is determined that the error value 38 of the sub-set of primary images 30 is not the lowest so far, then the method 300 returns to block 302 to capture at least one further primary image 30 and to identify another later sub-set of primary images 30.
• the later sub-set of primary images 30 may be offset from the earlier sub-set of primary images 30 by one temporally adjacent primary image 30.
• Each sub-set of primary images 30 is offset from a preceding sub-set of primary images 30 by one temporally adjacent primary image 30. In other examples, there may be an offset of more than one primary image between sub-sets of primary images.
• the method 300 continues to block 310 in which ambient light correction is performed on the error- minimised sub-set of primary images 40 to form an ALC image 42 having a minimised error.
• the third example method 300 is very similar to the first example method 100, and differs in that it estimates the error value of sub-sets of primary images 30 in real time on temporally consecutive sub-sets of primary images, and performs ambient light correction on a sub-set of primary images 30 each time the respective sub-set has an estimated error value 38 lower than previously estimated error values, rather than determining which of a plurality of sub-sets of primary images 30 has the lowest estimated error after the plurality of primary images 30 has been captured, and performing ambient light correction on the error minimised sub-set.
• a computer program may be stored or distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Signal Processing (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Image Analysis (AREA)
• Image Processing (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=74871201

Family Applications (1)

Country Status (5)

Cited By (1)

Citations (3)

Family Cites Families (17)

• 2021
  • 2021-03-11 EP EP21161944.0A patent/EP4057217A1/en not_active Withdrawn
• 2022
  • 2022-01-26 WO PCT/EP2022/051735 patent/WO2022189060A1/en not_active Ceased
  • 2022-01-26 EP EP22705020.0A patent/EP4305589B1/en active Active
  • 2022-01-26 US US18/276,288 patent/US12526534B2/en active Active
  • 2022-01-26 JP JP2023554860A patent/JP7612890B2/ja active Active
  • 2022-01-26 CN CN202280020391.6A patent/CN116964621A/zh active Pending

Patent Citations (3)

Also Published As

Similar Documents

Legal Events