Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/11—Region-based segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/40—Analysis of texture
  • G06T7/41—Analysis of texture based on statistical description of texture
  • G06T7/44—Analysis of texture based on statistical description of texture using image operators, e.g. filters, edge density metrics or local histograms
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
  • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/50—Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
  • G06V10/507—Summing image-intensity values; Histogram projection analysis
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20112—Image segmentation details
  • G06T2207/20132—Image cropping

Definitions

• This invention relates to image modelling in an image transmission system.
• the invention is applicable to, but not limited to, image modelling using multiple parameters to identify anisotropic features.
• Future generation mobile communication systems are expected to provide the capability for video and image transmission as well as the more conventional voice and data services. As such, video and image services will become more prevalent and improvements in video/image compression technology will likely be needed in order to match the consumer demand within available bandwidth.
• the interpretation data may or may not be transmitted in compressed form.
• Model-based techniques tend to produce good results where the range of objects (more generally content) of interest is limited, but where the environment is relatively unconstrained. Image-driven approaches perform better in a relatively constrained environment. In practice, most systems tend to combine elements of the two approaches .
• the image-driven approach relies on features in the image, such as edges or corners, to propagate "naturally" and form meaningful descriptions or models of image content.
• a typical example is x figure-ground' image segmentation, where the task is to separate the object of interest in the foreground from the background.
• a number of small salient patches or 'icons' are identified within an image. These icons represent descriptors of areas of interest.
• saliency is defined in terms of local signal complexity or unpredictability or, more specifically, the entropy of local attributes. Icons with a high signal complexity have a flatter intensity distribution and, hence, a higher entropy. Examples of intensity distributions are shown in figure 1, which is discussed later. In more general terms, it is the high complexity of any suitable descriptor that may be used as a measure of local saliency.
• Known salient icon selection techniques measure the saliency of icons at the same scale across the entire image.
• the particular scale selected for use across the whole image may be chosen in several ways. Typically, the smallest scale, at which a maximum (peak) occurs in the average global entropy, is selected.
• the size of image features varies. Therefore, a scale of analysis, that is optimal for a given feature of a given size, might not be optimal for the same feature of a different size.
• scale information is important in the characterisation, analysis, and description of image content. For example, prior to filtering an image, it is necessary to specify the kernel size, or in other words the scale, of a filter to use as well as the frequency response. It is also known that filters are commonly used in image processing for tasks such as edge-detection and anti-aliasing.
• scale can be regarded as a measurement to be taken from an image (region) , and hence can be used as a descriptor.
• the description extracted from the image may be used for subsequent matching or classification of that image (region) .
• One example may be in segmenting parts of an aerial image into different regions.
• the method In order to extract an appropriate scale description, the method must capture the scale behaviour of the most 'dominant', or in other words the most salient, scales in an image .
• Recent work in the area of matching and registration of aerial images has shown some promising results using a purely image-based approach. This is described in the paper: "Robust Description and Matching of Images. PhD thesis, University of Oxford, 1998", by Sebastien Gilles.
• Salient icons are selected by an algorithm in each of the images to be registered and, by matching these; the approximate global transform between the images may be determined.
• Gilles' experiments with a global image matcher show that such (global) methods are prone to failure due to local minima in the optimisation function.
• Gilles' approach is to use the local salient icon matcher to find the approximate transform, and then to use the global technique to fine-tune this.
• a focus of the present invention is to define how those areas of an image (or image sequence) are selected to be representative of the image content (or salient) . Hence, the selected areas can be used as the source for efficient description or content interpretation. Gilles defines saliency in terms of local signal complexity or unpredictability; more specifically he suggests the use of the entropy of local attributes.
• R x is the probability of descriptor 'd' taking on value 'di' in the local region ⁇ R x ' .
• the inventors of the present invention have recognised a problem with the Gilles technique, in that it is necessary to select a scale or size of the local neighbourhood region for the entropy calculation and subsequent description.
• Gilles' original algorithm measures the saliency of image patches at the same scale across the entire image. Since the size of image features vary, any feature identification technique needs to account for this by selecting an optimum scale (or set of scales) of analysis for a given feature.
• the Gilles method considers only the feature-space saliency of an image, whereas the analysis should include the scale dimension as well. That is, the measure of saliency should measure saliency over scale as well as feature-space.
• FIG. 2 is a flowchart illustrating the known method for determining icon salience, as described in UK patent application GB-A-02367966.
• a region defined by a scale s is typically initialised to a minimum scale value s in step 200.
• the region is defined around a pixel at coordinates (x, y) in an image, as shown in step 210.
• PDF probability density function
• the values may be grey- level intensity values, colour levels, or any other characteristic used to define the type of interest desired for image features.
• a Parzen window PDF estimator for example using a Gaussian kernel method or a basic histogram binning method may be used as the PDF estimation algorithm.
• the entropy H D of region (x, y, s) is then calculated in step 230.
• Scale s is then increased by the next scale increment in step 240, and the previous steps are repeated for all scales of s between Si and a maximum scale value s in step 250.
• Steps 200 to 270 are then preferably repeated for all pixels in the image, as shown in step 280.
• the entropy values of regions of peak entropy have been weighted, they are preferably ranked by weighted peak entropy value, thus resulting in a ranking of the pixels by peak saliency, in step 290.
• peak width is used as a weighting factor in the calculation of saliency.
• the measure for self-similarity that is used is the sum of absolute difference in the histogram of the local descriptor, although there are many alternative ways in which two PDFs may be compared (for example Kullback- Leibler) .
• the calculation in the continuous case is as follows :
• equation [3.1] For a practical implementation, the discrete case of equation [3.1] is required, as shown in equation [3.2] :
• W D The saliency over scale measure
• equation [4 . 1 ] is required, as shown in equation [4 . 2 ] :
• the absolute partial derivative term is approximated by the sum of absolute difference in the discrete case.
• the vector of scales at which the entropy peaks, s p is defined by:
• the method generates a 3D space (2 spatial dimensions plus scale) sparsely populated by scalar saliency values.
• the above definition does not specify how the probabilities, p(d,s,x), are to be obtained from the image.
• a circular sampling window is suggested. This is beneficial because it enables a rotationally invariant saliency measure.
• any suitable single parameter-sampling window may be used.
• the inventors of the present invention have recognised that the restriction in the prior art arrangements to a single-parameter scaling function, for example the radius of a circle, biases the method towards isotropically salient features. Furthermore, the method cannot measure local orientation. Also, sub-optimal scales are selected for features exhibiting anisotropy, for example, short sections of line-like features.
• an image transmission device as claimed in Claim 11.
• an image transmission unit as claimed in Claim 14.
• a storage medium storing processor-implementable instructions for controlling a processor to carry out any of the aforementioned method steps of the first and/or second aspect of the present invention, as claimed in Claim 16.
• inventive concepts of the present invention overcome the limitations of the prior art approaches by providing N-parameter sampling windows to account for anisotropic features/regions .
• the inventors propose to generalise the isotropic, single parameter sampling function to N- parameter sampling windows to account for anisotropic features/regions. In this manner, a better description of the image, and therefore a feature in the image, is determined.
• FIG. 1 illustrates a series of local histograms of intensity of an image, useful in understanding the context of the present invention
• FIG. 2 is a flowchart illustration of a known method for determining icon salience.
• FIG. 3 shows an elliptical function that provides an example of a plurality of parameter functions that can be used in identifying a feature in an image, in accordance with the preferred embodiment of the invention
• FIG. 4 shows a flowchart for identifying a feature using a plurality of parameters, in accordance with an enhancement to the preferred embodiment of the invention
• FIG. 5 illustrates a device for identifying a feature in an image, in accordance with an embodiment of the invention
• FIG. 6 illustrates salient regions identified using the known isotropic scale saliency method of FIG. 2
• FIG. 7 illustrates salient regions identified using the anisotropic scale saliency method of FIG. 4, in accordance with an embodiment of the invention
• FIG. 8 illustrates an original view of a cheetah
• FIG. 9 illustrates a stretched version of the cheetah of FIG. 8 to indicate how the method of FIG. 4 can be used for matching across different views of an object.
• the preferred embodiment of the present invention overcomes the limitations of the aforementioned approaches by providing N-parameter sampling windows to account for anisotropic features/regions.
• this approach is able to deal with some of the effects of projective distortion, for example, 3D effects such as perspective.
• 3D effects such as perspective.
• a practical benefit of this approach is that the generalised Scale Saliency algorithm is able to select an improved set of features and scales in cases where the image is of a 3D scene. For example, in an orthographic view, a circle is imaged as a circle. However, in another view, the circle might map onto an ellipse.
• a yet further benefit of the generalised concept is that orientation information can now be captured.
• the multiple parameter function uses scale, axis ratio, and orientation to parameterise more accurately the ellipse.
• the W D value has only to be calculated for one parameter instead of two. Indeed, a better parameterisation might be to use scale (s) , ratio
• the preferred embodiment of the present invention is invariant to anisotropic scaling and shear; that is, the full Affine set of transformations.
• the Affine set is a first order approximation to the full projective set of transformations, as known to those skilled in the art. Consequently, it will not be further described here.
• the preferred enhanced method of the present invention is described below with reference to the flowchart 400 of FIG. 4A and FIG. 4B.
• the flowchart focusses on calculating the entropy for regions of pixels, identifying peaks and applying a weighting function.
• the enhanced process is carried out for each pixel in the image, where the pixel position is identified by 'x' and 'y' co-ordinates, as shown in step 402.
• the rotation (angle) step 'a' of the elliptical function is set to zero, as in step 404.
• the variables 'rmin' and 'ratiodiv' specify a number and range of ratios to be tried.
• the scale value S is then set to a minimum (Smin) , as in step 408, and as described in UK Patent Application GB-A-02367966.
• the image sampler samples the local descriptor values at image location I(x,y) using the aforementioned values for 'a', 'r' and 's', as shown in step 410.
• the preferred arrangement for generating these samples is also described in co-pending UK Patent Application GB-A- 02367966 filed by the same applicant.
• a probability density P(d,s) value is set to an estimate of the local probability density function (PDF) from the IS samples, as shown in step 412.
• PDF local probability density function
• the PDF calculation can be performed by any known mechanism, such as a histogram technique, as described in UK Patent Application GB-A-02367966.
• the entropy (H D ) for that sample is then calculated, for each of the aforementioned values for 'a', 'r' and 's', as shown in step 414.
• the scale saliency (W D ) for that sample is also calculated for the aforementioned values for 'a', 'r' and 's', as shown in step 416.
• step 418 The process then moves on to the next scale sample, as in step 418. If the next scale sample is not greater than the maximum scale sample in step 420, the above process repeats from step 410 through step 418.
• the application of S ma ⁇ is described in UK Patent Application no. GB-A- 02367966. If the next scale sample is greater than the maximum scale sample S ma ⁇ , in step 420, the ratio step is incremented in step 422.
• step 424 If the next ratio value is not greater than '1' in step 424, the above process repeats from step 408 through to step 422. That is, the scale value for the next ratio value is set to a minimum, and a new set of scale values for the new ratio calculated.
• step 424 If the next ratio value is greater than '1', in step 424, the angle value 'a' is incremented in step 426, in accordance with the selected angle step-size.
• the number of angle steps ( 'number_angles' ) may be selected in order to perform a predetermined number of samples between 0 and ⁇ . A higher number of angle steps will give a more accurate result, but will of course be slower.
• step 428 For this new angle value, a determination is made as to whether the maximum angle ' ⁇ ' has been sampled, in step 428. If the maximum angle ' ⁇ ' has not been sampled in step 428, the above process repeats from step 406 through step 426. That is, the scale value and the ratio value are both set to a minimum, and new sets of scale values and ratio values are calculated for this new angle.
• the flowchart 400 then moves on to FIG. 4B, once calculations for each value of entropy (H D ) , local descriptor (IS), PDF (P(d,s)) and scale saliency (W D (a,r,s) have been completed for each value of angle, ratio and scale (a,r,s) .
• a filter preferably a three- tap averaging (smoothing) filter is applied to the scale saliency values (W D (a,r,s)), as shown in step 430.
• This smoothing step is applied with respect to the scale parameter only, for all angles and ratios, and alleviates some of the potential noise problems.
• the angle step 'a' of the elliptical function is set to zero in step 432.
• the variables 'r m i n ' and 'ratiodiv' specify a number and range of ratios to be tried.
• the scale value S is then set to a minimum (S m i n +1) in step 436.
• step 452 For this new angle value, a determination is made as to whether the maximum angle ' ⁇ ' has been sampled, in step 452. If the maximum angle ' ⁇ ' has not been sampled in step 452, the above p'rocess repeats from step 434 through step 448. That is, the scale value and the ratio value are both set to a minimum, and new sets of scale values and ratio values are calculated for this new angle. If the maximum angle ' ⁇ ' has been sampled in step 452, the process ends by determining the feature (isotropic or anisotropic) of the image in step 454.
• FIG. 4A and FIG. 4B describes replacement of the single parameter sampling function with a multiple parameter version. In this case, it's a three-parameter version. The entropy is calculated at each value of each of the three parameters .
• the scale saliency (W D ) measure is modified such that the partial derivative is taken with respect to only the scale parameter.
• the inventors have recognised that the rotation angle and ratio cannot be used in this context, as they would measure unpredictability in the rotation angle and ratio. It is not viable to look for peaks over a rotation dimension, as that would bias the method against round items, such as circles, which are not affected by rotation. It is inefficient to look for peaks with respect to ratio, as these do not add to the saliency decision.
• the peak detection is similarly modified to search for peaks in entropy with respect to scale, but not rotation angle or ratio. The correct orientation is still found because the shape that causes the largest inter-scale saliency measure (W D ) is the one that matches the feature shape .
• the vector of scales at which the entropy peaks, s p becomes a matrix, S p with three rows, one for each of the scale variables and as many columns as peaks at that position.
• the modified equations are as follows:
• the invention has been described above in terms of a method. However, the invention also comprises an image transmission unit/device functioning in accordance with the invention.
• FIG. 5 An embodiment of such a device is illustrated in FIG. 5.
• the device of FIG. 5 serves to rank pixels in an image by degrees of saliency.
• the device comprises a processor 500 providing a defining function 510 that defines a plurality of regions for each of a plurality of pixels in an image 505.
• each of the regions is defined over a different scale 's', at a ratio 'r' and angle 'a' for any one of the pixels at coordinates (x, y) .
• the processor also includes a calculation function 530 for calculating an entropy value for each of the regions, for each permutation of the respective variables a, r, s .
• the processor also includes a peak identification function 560 for identifying any peak entropy regions.
• the peak entropy regions are regions that include a peak entropy value and a weighting function 570 for weighting the peak entropy value of each of the peak entropy regions by a weight value corresponding to a peak width estimate of the peak entropy value. This is described in UK patent application GB-A-02367966.
• the processor includes a feature identification function 590 for identifying a feature in an image based on the determined peaks.
• the focus of the preferred embodiment is to find interesting regions in an image in an unsupervised manner.
• the weighted saliency measure supplies values about how interesting something is.
• the present invention focuses on identifying a feature in an image, which to all intent and purpose is such an interesting region in an image and the two expressions should be deemed synonymous .
• FIG. 5 Also shown in FIG. 5 is an 'acquisition and frame structure' unit 508, which includes a camera for scanning the image. Such a unit is known to the person skilled in the art, and hence is not described in further detail.
• the PDF estimator 520 provides the probability density function, as described above with reference to the methods of the invention.
• FIG. 6 an example of the performance of the original isotropic scale saliency algorithm is illustrated.
• FIG. 7 illustrates an example of the performance of the modified anisotropic scale saliency algorithm. Comparison of the two performances of FIG. 6 and FIG. 7 highlights that the anisotropic version correctly identifies the scales of the ellipses and the circle. In contrast, the isotropic version correctly detects only the circle and finds numerous features along the ellipses.
• any multiple-sampling window or alternative parameterisations instead of parameterising an ellipse using a major/minor axis ratio, a rotation, and a scale, it is possible to use two scales and a rotation.
• any polygon with N sides can be parameterised using N-l terms (for example the corner angles), a rotation and a scale. The W D term should then be modified to calculate the partial derivative with respect to each of the parameters.
• the video or image communication device may select salient regions of an image for preferential transmission. Particularly where the transmission is over a radio link to or from a mobile or portable radio (PMR) or mobile telephone, the transmission may comprise the selected regions only. Alternatively, these regions may be transmitted more frequently than other regions.
• the video or image communication device may form part of a mobile or portable radio (PMR) or cellular telephone.
• PMR mobile or portable radio
• the image transmission system, transmission unit and method for identifying a feature within an image provides at least the following advantages: (i) Extending the ability to find interesting regions of an image, as described in UK Patent Application GB-A- 02367966, to anisotropic regions.
• scale definition is extended to multiple parameters, thereby generalising the structures that can be identified, (ii) If an ellipse is parameterised using scale, rotation and a ratio of the major and minor axes, the method is invariant to anisotropic scaling and shear. As such, it is compatible with the full Affine set of transformations .
• FIG. 8 illustrates an original view of the Cheetah
• FIG. 9 illustrates a stretched version. It is noteworthy that the interesting regions (the spots) are still correctly identified in the stretched version of FIG. 9, by utilising the aforementioned inventive concepts.
• a method for identifying a feature in an image includes the steps of defining a plurality of saliency values of an image or set of images over multiple parameters, and calculating an entropy value for each of the saliency values over substantially each of the multiple parameters.
• the method further includes the steps of determining one or more peak entropy values for the image or set of images based on the calculated entropy values; and identifying a feature of said image or set of images based on the determined peak entropy value .
• an image transmission device includes a processor having the following functions: a defining function for defining a plurality of saliency values of an image or set of images over multiple parameters; a calculation function for calculating entropy value for each of the saliency values over substantially each of the multiple parameters; a peak entropy identification function for identifying any peak entropy value for the image or set of images based on said calculated entropy values; and a feature identification function identifying a feature in the image or set of images based on the peak entropy value .
• An image transmission unit adapted to perform the above method steps has also been provided.
• an image transmission system adapted to facilitate any of the above method steps or incorporate the above image transmission device has been provided.
• a storage medium storing processor-implementable instructions for controlling a processor to carry out any of the above method steps has been provided.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Multimedia (AREA)
• Probability & Statistics with Applications (AREA)
• Image Analysis (AREA)
• Apparatus For Radiation Diagnosis (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=9943942

Family Applications (1)

Country Status (4)

Families Citing this family (2)

Family Cites Families (2)

• 2002
  • 2002-09-13 GB GB0221175A patent/GB2393062B/en not_active Expired - Lifetime
• 2003
  • 2003-06-10 EP EP03794831A patent/EP1540595A1/de not_active Withdrawn
  • 2003-06-10 WO PCT/EP2003/006115 patent/WO2004025570A1/en not_active Ceased
  • 2003-06-10 AU AU2003242670A patent/AU2003242670A1/en not_active Abandoned

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events