US10521882B2 - Determination of brightness values of virtual pixels - Google Patents

Determination of brightness values of virtual pixels Download PDF

Info

Publication number
US10521882B2
US10521882B2 US15/737,785 US201615737785A US10521882B2 US 10521882 B2 US10521882 B2 US 10521882B2 US 201615737785 A US201615737785 A US 201615737785A US 10521882 B2 US10521882 B2 US 10521882B2
Authority
US
United States
Prior art keywords
pixels
filter coefficients
local
image processing
interpolation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US15/737,785
Other languages
English (en)
Other versions
US20190005617A1 (en
Inventor
Jörg Kunze
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Basler AG
Original Assignee
Basler AG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Basler AG filed Critical Basler AG
Assigned to BASLER AG reassignment BASLER AG ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KUNZE, JORG
Publication of US20190005617A1 publication Critical patent/US20190005617A1/en
Application granted granted Critical
Publication of US10521882B2 publication Critical patent/US10521882B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T5/008—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20004—Adaptive image processing
    • G06T2207/20012—Locally adaptive
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20092—Interactive image processing based on input by user

Definitions

  • the invention relates to an image processing apparatus for processing image data of an image sensor with a regular arrangement of first pixels, as well as to a digital camera, which comprises an image sensor and the image processing apparatus.
  • the invention further relates to a corresponding image processing method as well as a computer apparatus and a computer program product.
  • FIG. 1 shows schematically and exemplarily the structure of a digital camera 10 with a lens 22 .
  • a scene 30 is projected via the lens 22 onto an image sensor 31 , which comprises a regular arrangement of light-sensitive elements, so-called pixels.
  • the image sensor 31 conveys electronic data to a processing unit 32 , which usually is located in the camera 10 and which, e.g., comprises a processor, a digital signal processor (DSP), or a so-called field programmable gate array (FPGA). It can thereby be necessary to convert analog image data into digital image data, e.g., by means of an analog-to-digital converter (not shown in the figure). Where applicable, further desired mathematical operations, e.g., a color correction or a conversion into another image format, are executed on the image data in the processing unit 32 before the data are subsequently output as an electronic signal 34 via an interface 33 .
  • DSP digital signal processor
  • FPGA field programmable gate array
  • the present inventor has considered it an object to develop a means to generate, from an image of an image sensor with a first pixel size, an image with a second pixel size, which preferably is freely selectable and differs from the first pixel size, without the need of using a different image sensor type with the second pixel size.
  • a different image sensor type with the second pixel size.
  • more than one application with more than one requirement concerning the pixel size and number could be addressed with the same camera hardware with the same image sensor type. This may allow to save at least part of the costs for the forming of the different variants and/or to reduce the respective organizational effort.
  • a known method for generating in an image sensor with a first pixel size an image with a second pixel size is known as “binning”, wherein usually a distinction is made between (1) charge domain binning, (2) voltage domain binning, and (3) digital domain binning.
  • the first case comprises the combining of charge packets of neighboring pixels or pixels that are arranged in spatial proximity
  • the second case combines voltage signals
  • the third case combines digital signals, whereby a charge, voltage or digital value is obtained, which represents the signal of a so-called “superpixel” with a second pixel size.
  • Such binning methods are described in detail, e.g., in the European patent document EP 0 940 029 or in the US patent document U.S. Pat. No. 6,462,779.
  • FIG. 2 shows schematically and exemplarily different binning processes.
  • FIG. 2( a ) shows an image sensor with a regular arrangement of pixels 40 with a first pixel size.
  • FIG. 2( b ) shows an arrangement of superpixels 41 , which were generated by binning horizontally neighboring pixels, that are twice as wide and have the same height as the pixels 40 .
  • a binning is called a 2 ⁇ 1 binning.
  • FIG. 2( c ) shows an arrangement of superpixels 42 , which were generated by binning horizontally and vertically neighboring pixels, that are twice as high and twice as wide as the pixels 40 .
  • such a binning is called a 2 ⁇ 2 binning.
  • the visible spaces between the superpixels 41 in FIGS. 2( a ) to ( c ) only serve to make their outlines more easily recognizable and are not present in reality.
  • binning can be used to generate superpixels with sides lengths that are an integer multiple of the side lengths of the first pixels, respectively. It is, however, possible that a second pixel size is desired with side lengths that are not an integer, e.g., a rational, multiple of the side lengths of the first pixels. Such an aspect ratio cannot be realized by means of binning methods so far.
  • FIG. 3 An example for a non-integer, rational multiple of the side lengths of the second pixels relative to the side lengths of the first pixels is schematically and exemplarily shown in FIG. 3 .
  • the second pixels 44 have side lengths of 6/5 of the side lengths of the first pixels 43 .
  • the visible spaces between the second pixels 44 are not present in reality and only serve to make their outlines more easily recognizable.
  • a further disadvantage of charge domain binning is that it can only be realized in such image sensors, in which this is possible due to their design, e.g., in a number of CCD sensors, but not in image sensors in which it is not possible due to the design, e.g., in various CMOS sensors.
  • interpolation methods are also known, in which the resolution of an image is changed by means of an interpolation, such as it is described in detail in the US patent documents U.S. Pat. Nos. 7,567,723 and 7,286,721.
  • interpolation methods are, e.g., the so-called nearest neighbor interpolation, the bilinear interpolation, as well as the bicubic interpolation (see, e.g., FIG. 4 of U.S. Pat. No. 7,567,723).
  • the resolution of the image data of an image sensor can be changed.
  • the image data which have been changed in their resolution in this manner, do not have characteristics that correspond to the image data that would be expected from an image sensor with a correspondingly changed pixel size and number.
  • the interpolation so to speak, provides a brightness value of a virtual output pixel (second pixel) at a different location than the input pixels (first pixels) that form the input image.
  • an image processing apparatus for processing image data of an image sensor with a regular arrangement of first pixels, wherein the image processing apparatus is configured to determine a brightness value for each of two or more virtual second pixels of the same size at different intermediate positions between the first pixels, wherein the determination of the respective brightness value comprises an interpolation of the pixels of a neighborhood of the respective intermediate position, the neighborhood comprising several of the first pixels, by means of an associated local filter, wherein each of the local filters comprises a plurality of filter coefficients, wherein for at least one of the local filters more than one of the filter coefficients is unequal to zero, and wherein the sum of the squared filter coefficients for each of the local filters is equal to a constant value, which according to a first condition is the same for all local filters.
  • FIG. 4 shows schematically and exemplarily the effect of a known interpolation on the characteristics of a pixel.
  • a signal value c for an output pixel 82 is determined as a result from the left input pixel 80 with a signal value a and the right input pixel 81 with a signal value b.
  • the small deviations in the height of the pixels 80 , 81 , 82 , and 83 shown in FIG. 4 are not present in reality and only serve to make their outlines more easily recognizable in the representation. That is, contrary to the representation in FIG. 4 , the pixels 80 , 81 , 82 , and 83 have an identical extent in the vertical direction.
  • a+b of the signal values of the pixels 80 and 81 the signal value of a superpixel 83 is formed, the area of which comprises the areas of the pixels 80 and 81 and which therefore comprises an area twice as large as the input pixels.
  • the signal value of this twice-as-large superpixel 83 is multiplied with the gain factor 1 ⁇ 2, such that the correct brightness is obtained.
  • the linear interpolation in this case leads to the result that the output pixel comprises a twice as large area compared to the input pixels and a gain factor that is half as large.
  • the first image was bilinearly interpolated to generate a second image with a resolution of 31/32 of the original image, i.e., with a resolution of 1984 ⁇ 1984 pixels (second pixels).
  • a standard deviation of 32 should still result while maintaining the image energy.
  • the results show a considerable decrease of the noise to values that periodically oscillates between 18 DN and 26 DN.
  • This decrease of the noise is indicative of a corresponding decrease of the image energy, which is perceived by a viewer as a disadvantageous loss of image sharpness.
  • This decrease shows a spatial periodicity which is disturbing when the image is further mathematically analyzed, e.g., according to the EMVA standard 1288.
  • the determination of the respective brightness value comprises an interpolation of the pixels of a neighborhood of the respective intermediate position, the neighborhood comprising several of the first pixels, by means of an associated local filter, wherein each of the local filters comprises a plurality of filter coefficients, wherein for at least one of the local filters more than one of the filter coefficients is unequal to zero, and wherein the sum of the squared filter coefficients for each of the local filters is equal to a constant value, which according to a first condition is the same for all local filters.
  • the local filters are preferably linear filters. Since the virtual second pixels shall adhere as much as possible to a linear pixel model according to the EMVA standard 1288, the interpolation is preferably performed, under the assumption that also all first pixels adhere to the linear pixel model according to the EMVA standard 1288, by means of a mathematical operation that itself is linear and that, moreover, can be formulated as a filtering. This linear filtering is realized for each of the different intermediate positions by means of an associated local filter with a plurality of filter coefficients. Since for at least one of the local filters more than one of the filter coefficients is unequal to zero, a “real” interpolation can be realized, in which the brightness value of the virtual second pixels is determined on the basis of more than one of the first pixels.
  • the term “local” indicates that the filters used in the interpolation have a finite filter size. Only with a filter with a finite, i.e., limited, size, a result can be achieved with a finite processing time and/or a finite amount of resources.
  • the local filters are applied to the neighborhood of the respective intermediate position, which respectively comprises several of the first pixels.
  • the constant value corresponds to the square of a noise gain, wherein the noise gain corresponds to the product of a predetermined gain of the virtual second pixels relative to a gain of the first pixels and the reciprocal of the square root of a predetermined relative pixel size (w), when the relative pixel size corresponds to the ratio of the size of the virtual second pixels to the size of the first pixels.
  • the transmission of the noise not only occurs in a spatially homogeneous manner, but the gain value for the transmission of the noise also corresponds to the desired size of the output pixels (second pixels).
  • gain is to be understood in this context such that it can also mean an attenuation of the noise, i.e., a change of the noise by a factor smaller than 1, or a constancy of the noise (factor equal to 1).
  • the second condition can be explained as follows:
  • the statistics of the photons and photoelectrons follows a Poisson distribution, respectively.
  • the mean values for the received photons ⁇ p and for the photoelectrons ⁇ e increase by the same factor w, independent of a gain g.
  • the mean values ⁇ p and ⁇ e are equal to the associated variances ⁇ p 2 and ⁇ e 2 , also these values increase by the factor w and, accordingly, the associated noise increases by the square root of w.
  • the gain of the input pixels (first pixels) and additionally a predetermined gain g (see also in the following) is applied to the noise
  • R noise gain
  • the term “relative pixel size” is understood here to describe a change of the area of the virtual second pixels relative to the area of the first pixels. For instance, if the virtual second pixels are 1.5 times as large as the first pixels both in the width and in the height, the relative pixel size w results to 1.5 ⁇ 1.5 equal to 2.25, i.e., the area of the virtual second pixels is enlarged by a factor w equal to 2.25 compared to the area of the respective first pixels.
  • the filter coefficients for each of the local filters additionally fulfill a third condition that the sum of the filter coefficients is equal to the predetermined gain.
  • the desired output brightness can be set relative to the input brightness.
  • the virtual second pixels exhibit a same convergence gain K according to the EMVA standard 1288 as the first pixels.
  • the gain should be selected such that it corresponds to the relative pixel size.
  • the output image virtual second pixels
  • the gain should be selected to be equal to 1.
  • the filter coefficients for each of the local filters additionally fulfill a fourth condition that the center of gravity of the filter coefficients corresponds to the associated intermediate position. Since the intermediate positions correspond to the center points of the virtual second pixels, respectively, geometrical distortions, which could otherwise result from the interpolation, can be avoided in this manner. Such a geometrical distortion would be, e.g., that a straight edge from the input image would not be recognizable in the output image as a straight edge anymore, but, e.g., as an annoying wave pattern.
  • the interpolation is a one-dimensional interpolation, wherein each of local filters is a one-dimensional local filter by means of which a one-dimensional interpolation is performed.
  • the resolution of the image data can be changed by means of the interpolation in a simple and efficient manner in one dimension, e.g., in the width or in the height.
  • the interpolation is a two-dimensional interpolation, wherein each of the local filters is a two-dimensional local filter by means of which the two-dimensional interpolation is performed, or wherein each of the local filters corresponds to the dyadic product of two one-dimensional local filters by means of which the two-dimensional interpolation is performed as two subsequently performed one-dimensional interpolations.
  • the resolution of the image data can be changed by means of the interpolation in two dimensions, e.g., in both the width and the height.
  • the size of the two-dimensional local filters the same in the two dimensions, or the size of the two one-dimensional filters in the respective one dimension is the same.
  • the image processing apparatus comprises a storage in which the filter coefficients of the local filters are stored for use in the interpolation.
  • the filter coefficients only have to be determined once in advance and stored in the storage. A repeated setting of the filter coefficients for each new filter process can therewith be avoided.
  • the image processing apparatus comprises an adjusting element for adjusting at least the relative pixel size, e.g., 1.5 ⁇ 1.5, if the virtual second pixels shall be 1.5 times as large as the first pixels both in the width and in the height, wherein the image processing apparatus is configured to set the filter coefficients of the local filters based on the adjusted relative pixel size.
  • the resolution of the output image is adjustable by means of the adjusting element.
  • the resolution does not follow automatically from the ratio of the size of the virtual second pixels to the size of the first pixels, but can be predetermined separately.
  • the digital camera e.g., firstly transmits images of low resolution with a high image refresh rate or with a low data rate, and only if an event is detected in the images, the same camera generates a high resolution image, in which additional details, e.g., a license plate, are recognizable.
  • the adjusting element can comprise. e.g., a control, such as a slide control or a rotary control, a register, a digital interface, or the like.
  • the setting of the filter coefficients comprises a determining of filter coefficients according to a synthesis function, wherein the determined filter coefficients preferably fulfill the first and the third condition, as well as a correcting of the determined filter coefficients or of the synthesis function based on a second and/or a third correction function in such a way that corrected filter coefficients are obtained that additionally fulfill the second condition and/or the fourth condition.
  • an image processing apparatus for processing image data of an image sensor with a regular arrangement of first pixels of a first pixel size for generating image data comprising virtual second pixels of a second pixel size, wherein the second pixel size is not an integer multiple of the first pixel size, wherein the image processing apparatus is configured to compute a brightness value for each virtual second pixel.
  • the image processing apparatus is the image processing apparatus according to one of claims 1 to 11 .
  • an image processing method for processing image data of an image sensor with a regular arrangement of first pixels determines a brightness value for each of two or more virtual second pixels of the same size at different intermediate positions between the first pixels, wherein the determination of the respective brightness value comprises an interpolation of the pixels of a neighborhood of the respective intermediate position, the neighborhood comprising several of the first pixels, by means of an associated local filter, wherein each of the local filters comprises a plurality of filter coefficients, wherein for at least one of the local filters more than one of the filter coefficients is unequal to zero, and wherein the sum of the squared filter coefficients for each of the local filters is equal to a constant value, which according to a first condition is the same for all local filters.
  • a computer apparatus comprising a processing unit that is configured to perform the image processing method as defined in claim 14 .
  • a computer program product comprises coding for causing a computer apparatus to perform the image processing method as defined in claim 14 , when the computer program product is executed on the computer apparatus.
  • FIG. 1 is a schematic, exemplary view of the structure of a digital camera
  • FIG. 2 is a schematic, exemplary view of two different binning processes
  • FIG. 3 is a schematic, exemplary view of an example of a non-integer, rational multiple of the side lengths of the second pixels relative to the side lengths of the first pixels;
  • FIG. 4 is a schematic, exemplary view of the effect of a known interpolation on the characteristics of a pixel
  • FIG. 5 is a schematic, exemplary view of the result of an experiment, in which an interpolation was applied to artificially generated noise images;
  • FIG. 7 is a schematic, exemplary view of a synthesis function for a linear interpolation
  • FIG. 8 is a schematic, exemplary view of an alternative synthesis function for a cubic interpolation
  • FIG. 9 is a schematic, exemplary view of a sign function as a second function for the correction of the position of the center of gravity
  • FIG. 10 is a schematic, exemplary view of a single sine wave as a second function for the correction of the position of the center of gravity;
  • FIG. 11 is a schematic, exemplary view of a spatially limited function, which is continuously differentiable at a border region, as a second function for the correction of the position of the center of gravity;
  • ) with ⁇ ⁇ /2;
  • ) mit ⁇ ⁇ /2 till;
  • ) mit ⁇ ⁇ /2;
  • FIG. 15 is a schematic, exemplary view of a synthesis function, which for a relative pixel size of w equal to 1 ⁇ 1 fulfills all four conditions;
  • FIG. 16 is a schematic, exemplary view of an alternative synthesis function, which for a relative pixel size of w equal to 5/3 ⁇ 5/3 fulfills all four conditions;
  • FIG. 17 is a schematic, exemplary view of a further alternative synthesis function, which for a relative pixel size of w equal to 2 ⁇ 3 ⁇ 2 ⁇ 3 fulfills all four conditions;
  • FIG. 18 is a schematic, exemplary view of a quantum efficiency curve in dependence of the selected pixel size w for a bicubic interpolation, determined according to the EMVA standard 1288;
  • FIG. 19 is a schematic, exemplary view of a quantum efficiency curve in dependence of the selected pixel size w for the inventive interpolation, determined according to the EMVA standard 1288.
  • FIG. 6 shows schematically and exemplarily a neighborhood, which comprises several of the first pixels 60 , of an intermediate position 62 for determining the brightness value of a virtual second pixel 61 through interpolation by means of a local filter.
  • the size of the neighborhood is 4 ⁇ 4 input pixels (first pixels) 60 for calculating the brightness value of the output pixel (virtual second pixel) 61 .
  • the neighborhood is selected such that the intermediate position 62 , which corresponds to the center point of the virtual second pixel 61 , is located in a central square 65 of the neighborhood.
  • the input pixels 62 are indexed by two counting variables i and j.
  • the relative position of the output pixel 61 is described, without limiting the generality, by the two position values x and y with respect to the selected neighborhood.
  • the value ranges here are ⁇ 1 ⁇ x ⁇ 2 and ⁇ 1 ⁇ y ⁇ 2.
  • the size of the neighborhood is equal to the size of the local filter.
  • the filter size is selected such that, on the one hand, the virtual second pixel 61 with the predetermined relative pixel size is completely located within the neighborhood and within the local filter, respectively, in the interpolation, and that, on the other hand, the resulting computational burden due to a too large selection of the neighborhood is not too large.
  • the determination of the respective brightness value comprises an interpolation of the pixels of a neighborhood of the respective intermediate position, the neighborhood comprising several of the first pixels 60 , by means of an associated local filter, wherein each of the local filters comprises a plurality of filter coefficients, wherein for at least one of the local filters more than one of the filter coefficients is unequal to zero, and wherein the sum of the squared filter coefficients for each of the local filters is equal to a constant value, which according to a first condition is the same for all local filters.
  • the local filters preferably also fulfill one or more of a second condition, a third condition, and a fourth condition, as also described above.
  • An interpolation function is selected for a predetermined filters size. This function provides for each intermediate position x or y (one-dimensional) or x and y (two-dimensional) of a virtual second pixel, respectively, the respectively associated filter coefficients values.
  • the intermediate position is only designated as x in the following, wherein x can be considered as being a vector in the case of a two-dimensional interpolation. If the filter coefficient values are only required for a finite number of intermediate positions x, they may also be provided in a table.
  • the required filter coefficients of the local filters for the interpolation can be determined with the help of a synthesis function, e.g., by sampling the synthesis function in a suitable manner.
  • This sampling is advantageously performed with intervals of 1 in relation to the size of the first pixels and, in particular, such that the center or zeropoint of the synthesis function is shifted to the value of x, respectively.
  • This is symbolized in the following explanations by describing the synthesis function in dependence of a parameter xf, wherein the values xf, at which the synthesis function is sampled for a particular intermediate position x, result to ⁇ x+n, wherein x assumes all negative or positive integer values including zero for which the value ⁇ x+n is in the value range of the synthesis function.
  • FIG. 7 shows schematically and exemplarily a synthesis function for a linear interpolation.
  • FIG. 8 An alternative synthesis function for a cubic interpolation is schematically and exemplarily shown in FIG. 8 .
  • the synthesis function for a linear interpolation according to FIG. 7 and the synthesis function for a cubic interpolation according to FIG. 8 are known from the prior art. They can be taken as a starting point in order to generate, on their basis, synthesis functions for an inventive interpolation by means of the corrections described in the following.
  • the interpolation function can be linear, piece-wise linear, quadratic, or cubic. In the two-dimensional case, it can satisfy, e.g., the known bilinear, bicubic, or spline interpolation.
  • the synthesis function continuously assumes the value zero at the border of its value range, which results from the predetermined filter size and which, e.g., in FIG. 7 ranges from ⁇ 1.0 to 1.0 and in FIG. 8 ranges from ⁇ 2.0 to 2.0, respectively. Therewith, jumps in brightness are particularly avoided when shifting the neighborhood.
  • the synthesis function is designed such that for all required intermediate positions x the sum of the associated filter coefficients obtained from the synthesis function is the same and, e.g., in the case that the relative pixel size w is equal to 1 ⁇ 1 and the gain g is equal to 1, assumes the value 1. Thereby, the third condition is fulfilled.
  • the fulfillment of the third condition can be achieved by means of a normalization.
  • a normalization One example of a function that does not fulfill the third condition is a Gaussian bell curve. To do so, the associated filter coefficients are divided for each intermediate position x by the previously determined sum of the filter coefficients.
  • This normalization can alternatively also be performed on the synthesis function itself. Since the normalization is performed in a point-wise manner, this can also lead to a change of the form of the curve of the synthesis function.
  • a correction can be performed. This correction can be achieved in that correction values are added to or multiplied with the synthesis function or the filter coefficients. Thereby, the correction can be performed in such a manner that the third condition, if it was fulfilled before, continues to be fulfilled.
  • a corresponding second function for an additive correction of the fourth condition can have the characteristic that it is neutral with respect to the third condition.
  • second correction function can be added to the synthesis function or to the filter coefficients without the third condition been violated thereby.
  • This neutrality is obtained as a result of the fact that for each intermediate position x, the sum of the values of the second correction function, which are associated to the respective filter coefficients values for the intermediate position x, amounts to zero. This is exactly the case when the sum of the values of the second correction function, when sampled with an equidistant spacing of 1, becomes zero, independent of the intermediate position x.
  • the second correction function has, for all intermediate positions x, the characteristic that a change of the center of gravity can be achieved by addition of this function to the synthesis function. This is achieved by the second correction function exactly in the case where the torsional moment of the sampling values (given an equidistant sampling with the spacing of 1) is unequal to zero.
  • the notion of the torsional moment of a local filter is defined in analogy to classical mechanics as the sum of the products of the filter coefficients with their respective positions in the synthesis function. Therewith, the last mentioned requirement becomes the statement that the torsional moment of the second correction function is unequal to zero for all intermediate positions x.
  • FIGS. 9 to 11 show schematically and exemplarily functions that can be used as a second function for a correction of the position of the center of gravity.
  • This function fulfills the above-described prerequisites for the second correction function.
  • it has the disadvantage that its spatial extent is not limited and that this therefore also applies to the spatial extent of the artifacts that may be generated by the function.
  • this function has the advantage that it continuously assumes a value of zero at the border of its effective range and, thus, that the spatial extent of the artifacts that may be generated by this function is limited.
  • a function may be mentioned that is continuously differentiable and becomes zero at the border of its effective range.
  • this function assumes the value zero, respectively.
  • this function has the advantage that it is continuously differentiable and becomes zero at the border of its effective range, wherefore the spatial extent of the artifacts that may be generated by the function is limited and visible “kinks” in the brightness profile are avoided.
  • the second condition can be achieved by addition of a multiple of a third correction function K3(xf).
  • a function is advantageously used that is neutral with respect to the third and fourth condition.
  • the fulfillment of the third and fourth condition is still maintained even if a multiple of the third correction function K3(xf) is added.
  • ) with ⁇ ⁇ .
  • This function is schematically and exemplarily shown in FIG. 13 and is modeled in its form according to the Ricker wavelet, which is also referred to in the literature as the Mexican hat function and which is often used as a sharpening filter for images.
  • the function that is proposed here has the advantage that it is neutral with respect to the third condition for all intermediate positions x, i.e., the sum of the associated filter coefficients is zero in each case. Compared to the function shown in FIG. 12 , it has the two advantages that it is continuous at the border of the window and is continuously differentiable.
  • ) with ⁇ ⁇ /2, which is schematically and exemplarily shown in FIG. 14 . Also this function is modeled in its form according to the Ricker wavelet. Compared to the function illustrated in FIG. 13 , however, it exhibits a slightly different frequency characteristic.
  • the second condition can be fulfilled by adding a—as the case may be, position-independent—multiple of the symmetric correction function E2(xf), which is free of a torsional moment and neutral with respect to the brightness, to the synthesis function F2(xf), which is also free of a torsional moment. Since the second condition is a special case of the first condition in which the constant value corresponds to the square of the noise gain R, it is therewith possible to obtain a synthesis function F3(xf) that fulfills all four conditions.
  • a value a is determined as the sum of the squared filter coefficients of E2(xf) for each intermediate position x
  • a value b is determined as twice the sum of the products of the respective filter coefficients of F2(xf) with E2(xf)
  • a value c is determined as the sum of the squared filter coefficients of F2(xf) minus R.
  • sqrt designates the square root function.
  • u2(xf) is the torsional moment of the filter coefficients of the function E1(xf)
  • v2(xf) is the torsional moment of the filter coefficients of the function K2(xf).
  • the function E2(xf) was obtained.
  • the associated filter coefficients of the respective local filter can be determined for each intermediate position x.
  • an interpolation can be performed such that the transmission of the noise not only occurs in a spatially homogeneous manner, but that also the gain value for the transmission of the noise corresponds to the desired area of the output pixels (second pixels).
  • An alternative synthesis function which for a relative pixel size of w equal to 5/3 ⁇ 5/3 fulfills all four conditions, is schematically and exemplarily shown in FIG. 16 . It was calculated from the same functions as in the above-described example. Only the relative pixel size w of the output pixels was changed to 5/3 ⁇ 5/3, i.e., the area of the virtual second pixels was enlarged by a factor w equal to 2.78 compared to the area of a respective first pixel. With the help of the filter coefficients obtained from this synthesis function, an image can thus be filtered such that the relative pixel size is 5/3 ⁇ 5/3 and, therewith, an enlargement of the pixels occurs both in the width and in the height. Also in this case, the transmission of the noise occurs in a spatially homogeneous manner, wherein the gain value for the transmission of the noise again corresponds to the desired size of the output pixels (virtual second pixels).
  • a further alternative synthesis function which for a relative pixel size of w equal to 2 ⁇ 3 ⁇ 2 ⁇ 3 fulfills all four conditions, is schematically and exemplarily shown in FIG. 17 . It was calculated from the same functions as in the above-described examples.
  • the relative pixel size w of the output pixels was changed to 2 ⁇ 3 ⁇ 2 ⁇ 3, i.e., the area of the virtual second pixels was reduced by a factor w equal to 2.25 compared to the area of a respective first pixel.
  • the filter coefficients obtained from this synthesis function, an image can thus be filtered such that the relative pixel size is 2 ⁇ 3 ⁇ 2 ⁇ 3 and, therewith, a reduction of the pixels occurs both in the width and in the height.
  • the transmission of the noise occurs in a spatially homogeneous manner, wherein the gain value for the transmission of the noise again corresponds to the desired size of the output pixels (virtual second pixels).
  • a second series of images with a corresponding second resolution was generated for different desired sizes of the virtual second pixels, respectively, using, on the one hand, a bicubic interpolation and, on the other hand, the inventive interpolation.
  • the ratio of the second resolution to the first resolution in the x and y direction corresponds to the reciprocal of the ratio of the size of the virtual second pixels to the size of the first pixels.
  • the resulting second series of images have been evaluated in the manner described in the EMVA standard 1288 under the assumption of the second pixel size.
  • FIG. 18 shows schematically and exemplarily the quantum efficiency values QE in percent obtained for the bicubic interpolation plotted against the relative pixel size w.
  • a relative pixel size 1 ⁇ 1 which is identical to the size of the first pixels, a correct quantum efficiency value 100 results.
  • a relative pixel size 2 ⁇ 2 which corresponds to an integer multiple of the size of the first pixels, also a correct quantum efficiency value 101 is obtained.
  • the resulting quantum efficiency values are significantly above 80%. These values are not correct. They do not correspond to the quantum efficiency of the digital camera, since, as a material and design dependent physical characteristic of the sensor, these should also stay constant when the size of the pixels is changed.
  • the determined QE values signify that the bicubic interpolation performs an undesired enlargement of the pixel area in a way that resembles what is shown in FIG. 4 .
  • FIG. 19 shows schematically and exemplarily the QE values for the inventive interpolation. Not only the quantum efficiency value 200 obtained for the original resolution, but also all other quantum efficiency values 201 substantially correspond to the correct value. This shows that the inventive interpolation correctly performs the change of the pixel size and that important physical characteristics of the pixels are correctly obtained when doing so.
  • the exemplarily mentioned local filters are substantially described in a one-dimensional way in order to allow for an easy understanding.
  • the corresponding teaching can also be applied to two-dimensional filters.
  • a two-, three-, or higher-dimensional effect can be obtained by subsequently performing multiple one-dimensional filter processes in an arbitrary order, e.g., first in the x direction, then in the y direction, et cetera.
  • the resolution of the image data of an image sensor shall be reduced by a factor of 5 ⁇ 6 in both the horizontal and vertical direction.
  • the size of the virtual second pixels is increased in comparison to the size of the first pixels in both directions by the factor 6/5 equal to 1.2, i.e., the relative pixel size w corresponds to 1.2, respectively, and in total 1.2 ⁇ 1.2 equal to 1.44 (i.e., ( 6/5) 2 ).
  • the intermediate positions can then be selected in each direction at the following positions between the first pixels: 2; 3.2; 4.4; 5.6; 6.8; 8; 9.2; 10.4; 11.6; 12.8; 14; et cetera.
  • the series of the intermediate positions does not start with zero but with 2, in order to provide for each virtual pixel a sufficiently large neighborhood for determining the respective brightness value.
  • the filter coefficients fulfill for each of the local filters the condition that their sum is equal to the predetermined gain g, i.e., equal to 1 (third condition), that their center of gravity corresponds to the associated intermediate position (fourth condition), and that the sum of their squares is equal to a constant value (first condition), which, here, is 0.833333 equal to 5 ⁇ 6.
  • This value corresponds to the square of the noise gain R, which corresponds to the product of the predetermined gain g of the virtual second pixels relative to the gain of the first pixels (i.e., 1 in this case) and the reciprocal of the square root of the predetermined relative pixel size w (i.e., here, the reciprocal of the square root of 6/5, i.e., the change of the size of the virtual second pixels relative to the first pixels in the respective direction). If the one-dimensional filter is applied in the horizontal and vertical direction, the desired reduction of the resolution by the factor 5/6 in both direction results.
  • the filter coefficients for each of the local filters fulfill the condition that their sum is equal to the predetermined gain g, i.e., equal to 1 (third condition), that their center of gravity corresponds to the associated intermediate position (fourth condition), and that the sum of their squares is equal to a constant value (first condition), which, here, is 0.694444 equal to (5 ⁇ 6) 2 .
  • This value corresponds to the square of the noise gain R, which corresponds to the product of the predetermined gain g of the virtual second pixels relative to the gain of the first pixels (i.e., 1 in this case) and the reciprocal of the square root of the predetermined relative pixel size w (i.e., here, the reciprocal of the square root of ( 6/5) 2 , i.e., the total change of the size of the virtual second pixels relative to the first pixels).
  • the filtering can be performed in a digital camera 10 , e.g., in the described calculating apparatus 32 .
  • the calculating apparatus 32 can be realized, e.g., as a processor, a micro processing unit (MPU) or a micro control unit (MCU), a digital signal processor (DSP), a field programmable gate array (FPGA), a graphics processing unit (GPU), or as an application specific integrated circuit (ASIC).
  • the calculating apparatus 32 can also be integrated into another module, for example, into the image sensor 31 or into the interface 32 . Therewith, it is, e.g., also possible to produce an image sensor 31 with a freely adjustable pixel size or a corresponding interface module 33 for a digital camera.
  • both the synthesis function and the filter coefficients are independent of the current image content, they do not have to be re-set for the calculation of each new second pixel. Rather, they can be set in advance and the filter values can be stored in a storage of the calculating apparatus 32 , e.g., in a table.
  • the mathematical operations that have to be performed during run-time are limited to the readout of the filter coefficients from a table and the application in the form of a linear filtering. This can be realized with the above-mentioned embodiments of the calculating apparatus 32 with only a small amount of computation time or with a small amount of resources.
  • the calculating apparatus 32 can also comprise an adjusting element for adjusting at least the relative pixel size (and, as the case may be, the gain of the virtual second pixels relative to a gain of the first pixels), wherein the calculating apparatus 32 is configured to set the filter coefficients of the local filters based on the adjusted relative filter size (and, as the case may be, the adjusted gain).
  • the resolution of the output image is adjustable by means of the adjusting element.
  • the filtering can also be performed after the output as an electronic signal 34 , e.g., in a portion of a technical system, in a computer, or in a smartphone.
  • a single unit or apparatus can perform the function of multiple elements that are recited in the claims.
  • the fact that individual functions and/or elements are recited in different dependent claims does not mean that a combination of these functions and/or elements could not be used to advantage.
  • an image processing apparatus for processing image data of an image sensor with a regular arrangement of first pixels, wherein the image processing apparatus is configured to determine a brightness value for each of two or more virtual second pixels of the same size at different intermediate positions between the first pixels, wherein the determination of the respective brightness value comprises an interpolation of the pixels of a neighborhood of the respective intermediate position, the neighborhood comprising several of the first pixels, by means of an associated local filter, wherein each of the local filters comprises a plurality of filter coefficients, wherein for at least one of the local filters more than one of the filter coefficients is unequal to zero, and wherein the sum of the squared filter coefficients for each of the local filters is equal to a constant value, which according to a first condition is the same for all local filters.
  • elements shown as integrally formed may be constructed of multiple parts or elements shown as multiple parts may be integrally formed, the operation of the interfaces may be reversed or otherwise varied, the length or width of the structures and/or members or connector or other elements of the system may be varied, the nature or number of adjustment positions provided between the elements may be varied.
  • the elements and/or assemblies of the system may be constructed from any of a wide variety of materials that provide sufficient strength or durability, in any of a wide variety of colors, textures, and combinations. Accordingly, all such modifications are intended to be included within the scope of the present innovations. Other substitutions, modifications, changes, and omissions may be made in the design, operating positions, and arrangement of the desired and other exemplary embodiments without departing from the spirit of the present innovations.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
US15/737,785 2015-12-15 2016-12-01 Determination of brightness values of virtual pixels Active US10521882B2 (en)

Applications Claiming Priority (7)

Application Number Priority Date Filing Date Title
DE102015121823.2 2015-12-15
DE102015121823 2015-12-15
DE102015121823 2015-12-15
DE102016113010.9 2016-07-14
DE102016113010 2016-07-14
DE102016113010.9A DE102016113010A1 (de) 2015-12-15 2016-07-14 Bestimmung von Helligkeitswerten virtueller Pixel
PCT/EP2016/079463 WO2017102361A1 (de) 2015-12-15 2016-12-01 Bestimmung von helligkeitswerten virtueller pixel

Publications (2)

Publication Number Publication Date
US20190005617A1 US20190005617A1 (en) 2019-01-03
US10521882B2 true US10521882B2 (en) 2019-12-31

Family

ID=58994225

Family Applications (1)

Application Number Title Priority Date Filing Date
US15/737,785 Active US10521882B2 (en) 2015-12-15 2016-12-01 Determination of brightness values of virtual pixels

Country Status (4)

Country Link
US (1) US10521882B2 (de)
EP (1) EP3391329B1 (de)
DE (1) DE102016113010A1 (de)
WO (1) WO2017102361A1 (de)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11582405B1 (en) 2021-09-23 2023-02-14 Qualcomm Incorporated Image data processing using non-integer ratio transforming for color arrays

Citations (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0532221A2 (de) 1991-09-04 1993-03-17 Westinghouse Electric Corporation Nichtganze Bildelementekompression für Vorwärts-Infrarotsensoren der zweiten Generation
EP0710925A2 (de) 1994-11-01 1996-05-08 International Business Machines Corporation System und Verfahren zur masstäblicher Umformung von Bildern
WO1997028641A1 (en) 1996-01-22 1997-08-07 California Institute Of Technology Active pixel sensor array with multiresolution readout
EP0969443A1 (de) 1998-06-29 2000-01-05 Sony Corporation Bilddatenverarbeitungsvorrichtung und -verfahren zum Verändern der Bildauflösung
US6295087B1 (en) * 1996-11-18 2001-09-25 Sony Corporation Image pickup apparatus having an interpolation function
US20020093510A1 (en) * 2001-01-15 2002-07-18 Asahi Kogaku Kogyo Kabushiki Kaisha Image interpolating device
US20020101525A1 (en) * 2001-01-26 2002-08-01 Asahi Kogaku Kogyo Kabushiki Kaisha Image interpolating device
US6462779B1 (en) 1998-02-23 2002-10-08 Eastman Kodak Company Constant speed, variable resolution two-phase CCD
US20040179747A1 (en) * 2003-03-11 2004-09-16 Canon Kabushiki Kaisha Image reading apparatus
US7286721B2 (en) 2003-09-11 2007-10-23 Leadtek Research Inc. Fast edge-oriented image interpolation algorithm
US7567723B2 (en) 2004-03-09 2009-07-28 Canon Kabushiki Kaisha Resolution changing method and apparatus
US7680359B2 (en) * 2001-11-19 2010-03-16 Stmicroelectronics, S.R.L. Method for merging digital images to obtain a high dynamic range digital image
US20100315536A1 (en) * 2005-06-03 2010-12-16 Hand Held Products, Inc. Method utilizing digital picture taking optical reader having hybrid monochrome and color image sensor
US20120002066A1 (en) * 2010-06-30 2012-01-05 Hand Held Products, Inc. Terminal outputting monochrome image data and color image data
US20130156346A1 (en) * 2010-07-09 2013-06-20 Samsung Electronics Co., Ltd. Image interpolation method and apparatus
US20150207974A1 (en) * 2014-01-17 2015-07-23 Texas Instruments Incorporated Methods and apparatus to generate wide dynamic range images

Patent Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0532221A2 (de) 1991-09-04 1993-03-17 Westinghouse Electric Corporation Nichtganze Bildelementekompression für Vorwärts-Infrarotsensoren der zweiten Generation
EP0710925A2 (de) 1994-11-01 1996-05-08 International Business Machines Corporation System und Verfahren zur masstäblicher Umformung von Bildern
WO1997028641A1 (en) 1996-01-22 1997-08-07 California Institute Of Technology Active pixel sensor array with multiresolution readout
EP0940029A1 (de) 1996-01-22 1999-09-08 California Institute Of Technology Aktivpixelsensormatrix mit mehrfachauflösungsausgabe
US6295087B1 (en) * 1996-11-18 2001-09-25 Sony Corporation Image pickup apparatus having an interpolation function
US6462779B1 (en) 1998-02-23 2002-10-08 Eastman Kodak Company Constant speed, variable resolution two-phase CCD
EP0969443A1 (de) 1998-06-29 2000-01-05 Sony Corporation Bilddatenverarbeitungsvorrichtung und -verfahren zum Verändern der Bildauflösung
US20020093510A1 (en) * 2001-01-15 2002-07-18 Asahi Kogaku Kogyo Kabushiki Kaisha Image interpolating device
US20020101525A1 (en) * 2001-01-26 2002-08-01 Asahi Kogaku Kogyo Kabushiki Kaisha Image interpolating device
US7680359B2 (en) * 2001-11-19 2010-03-16 Stmicroelectronics, S.R.L. Method for merging digital images to obtain a high dynamic range digital image
US20040179747A1 (en) * 2003-03-11 2004-09-16 Canon Kabushiki Kaisha Image reading apparatus
US7286721B2 (en) 2003-09-11 2007-10-23 Leadtek Research Inc. Fast edge-oriented image interpolation algorithm
US7567723B2 (en) 2004-03-09 2009-07-28 Canon Kabushiki Kaisha Resolution changing method and apparatus
US20100315536A1 (en) * 2005-06-03 2010-12-16 Hand Held Products, Inc. Method utilizing digital picture taking optical reader having hybrid monochrome and color image sensor
US20120002066A1 (en) * 2010-06-30 2012-01-05 Hand Held Products, Inc. Terminal outputting monochrome image data and color image data
US20130156346A1 (en) * 2010-07-09 2013-06-20 Samsung Electronics Co., Ltd. Image interpolation method and apparatus
US20150207974A1 (en) * 2014-01-17 2015-07-23 Texas Instruments Incorporated Methods and apparatus to generate wide dynamic range images

Also Published As

Publication number Publication date
EP3391329A1 (de) 2018-10-24
DE102016113010A1 (de) 2017-06-22
WO2017102361A1 (de) 2017-06-22
EP3391329B1 (de) 2019-09-18
US20190005617A1 (en) 2019-01-03

Similar Documents

Publication Publication Date Title
US9275463B2 (en) Stereo image processing device and stereo image processing method
JP4098319B2 (ja) 解像度変換方法および解像度変換装置
JP2018137789A (ja) ハイブリッド画像の分解および投影
US20150077573A1 (en) Projection system, image processing device, and projection method
CN107623844B (zh) 中间位置处的像素的颜色值的确定
EP2031861A2 (de) Verfahren zur Korrektur von Bildverzerrungen und Vorrichtung zur Bildverarbeitung dafür
JP2013225740A (ja) 画像生成装置、画像表示装置及び画像生成方法並びに画像生成プログラム
CN105230033B (zh) 利用自适应频率强度受控变换来创建图像中的细节
US9105106B2 (en) Two-dimensional super resolution scaling
JP5809865B2 (ja) 画像処理装置及び画像処理方法
TWI498852B (zh) 深度圖產生裝置及其方法
JP2015203952A (ja) 超解像装置及びプログラム
US10521882B2 (en) Determination of brightness values of virtual pixels
TWI544785B (zh) 影像縮減取樣裝置與方法
JP6305551B2 (ja) サブピクセルオフセットを有する電子ズーム方法
US20170345137A1 (en) Image processing apparatus and method for controlling the same
JP6378531B2 (ja) 超解像装置及びプログラム
JP2016115318A (ja) ぼやけ補正装置、超解像装置およびプログラム
JP6574682B2 (ja) コントラスト補正装置及びプログラム
JP2007035025A (ja) 画像変換装置、画像変換方法、プログラム及びコンピュータ可読記憶媒体
US20150085151A1 (en) Method and apparatus for fusing images from an array of cameras
JP2006238188A (ja) 補間フィルタおよび映像信号処理装置
JP6681774B2 (ja) コントラスト補正装置及びプログラム
JP6184277B2 (ja) 画像処理装置
JP2009147938A (ja) ビデオ映像形式の変換方法および対応する装置

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: BASLER AG, GERMANY

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KUNZE, JORG;REEL/FRAME:045696/0212

Effective date: 20180115

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE

CC Certificate of correction
MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4