EP0539653A2 - Detecting method of small objects in a natural environment - Google Patents

Abstract

A method and circuit arrangement for detecting small objects in a natural environment, using an electro-optical sensor, scanning the environment, followed by an evaluating unit and an image display device which may be provided. To be able to extract small compact objects out of their natural environment and to suppress irrelevant background information, a digital image is filtered by at least three non-linear filter branches operating in parallel, in such a manner that each filter determines the maximum and minimum of brightness values of in each case one of at least three pixel reference sets. The central pixel is marked in dependence on a threshold value and each marked pixel is enlarged by including the nearest adjacent pixels. The number of pixels marked is counted for a complete image. The current counting results are compared with fixed limits and used for controlling the threshold values. Finally, the enlarged marked pixels are combined by a logic AND operation to form a result set, and the coordinate values of all end result set elements determined in this manner are stored as detected pixels. <IMAGE>

Description

The invention relates to a method according to the preamble of claim 1 and a circuit arrangement for performing the method.

Every natural picture contains "pointed lights" which differ in contrast from their immediate surroundings. This is more or less true for many objects in the image, but smaller objects have certain symmetry properties that large objects do not. For example, a bright, straight branch of a tree in a picture against a dark background has bright neighboring pixels along the main axis of the branch and dark neighboring pixels perpendicular to it. In another example, a bright gas bubble in a liquid has dark neighboring pixels in all directions. Heuristically, the gas bubble has point symmetry and the branch has mirror symmetry. A whole tree in a landscape, on the other hand, has "little" symmetry, a star in the night sky, on the other hand, has a complete "point" symmetry. In principle, it therefore seems possible to discover small bubbles in a liquid environment as well as stars in the partially covered night sky.

The invention has for its object to provide a method of the type mentioned, with which small, compact objects in their natural environment (from image sequences) are extracted and almost all other irrelevant background information of the image is suppressed.

The object is achieved by the method steps indicated in the characterizing part of claim 1.

It is of particular advantage that the best possible extraction of small objects is achieved with the method according to the invention using symmetry principles and statistics without a great expenditure on circuitry. Process steps are specified by means of which the specific environment of each pixel in the image is analyzed. The surroundings are selected according to the different axes of symmetry of the objects to be detected. In practice, of course, the objects do not have perfect symmetry, in addition the backgrounds are noisy and very different. This creates a very different number of objects depending on the quality and content of the diversity of the images. In order to achieve a uniform mode of operation for different images, a suitable regulation is introduced so that the number of detected objects is limited to a statistically reasonable range.

Embodiments of the method according to the invention are described in subclaims 2 to 7.

• a) Das ankommende Bild wird zuerst durch ein zweidimenisonales FIR-Filter geschickt, um weißes Rauschen zu reduzieren.
• b) Das so gefilterte Bild wird weiter verarbeitet von vier nichtlinearen Filtern, die parallel arbeiten. Diese Filter wirken innerhalb eines Bereiches von z. B. 7 x 7 Bildpunkten (pixeln). Die Filter finden das Maximum und Minimum der Grauwerte einer Gruppe von Bildpunkten, die jeweils in den Ecken des 7 x 7 Bereiches (Matrix) angesiedelt sind. DIe spezielle Auswahl der verschiedenen Bildpunktgruppen ist aus Fig. 1 (schraffierte Dreiecke) zu entnehmen.
  Zur Vereinfachung soll die dreieckförmige Gruppe von Bildpunkten fortan Referenzmenge heißen. Dieser Verarbeitungsschritt berücksichtigt die Symmetrieerfordernisse zur Detektion kleiner, kompakter Objekte. Für zweidimensionale Objekte sind mindestens drei Referenzmengen und daher mindestens drei parallele Filterzweige erforderlich.
• c) Im nächsten Schritt werden in jedem Filterzweig die Ergebnisse der Maxima- und Minimaberechnungen mit dem aktuellen Bildpunkt(pixel) im Zentrum der Matrix verglichen.
  Eine Markierung des zentralen Bildpunktes als signifikant und ihr Eintrag in das Zwischenergebnisbild des Filterzweiges erfolgt, wenn die Kriterien
  
  $\text{P - M > T (K1)}$
  
  
  
  und
  
  $\text{M - m < L (K2)}$
  
  
  
  erfüllt sind.
  Dabei bedeuten:

  P:

  Intensität des zentralen Bildpunktes

  M:

  Maximum der Intensitäten der Referenzmenge

  m:

  Minimum der Intensitäten der Referenzmenge

  T:

  dynamischer Schwellwert

  L:

  fester Schwellwert

  Der Sinn dieses Verarbeitungsschrittes besteht darin, den zentralen Bildpunkt zu markieren, wenn sowohl der Bildpunkt genügend heller als seine Umgebung ist (dynamische Schwelle T) und seine Umgebung nicht zu stark strukturiert ist (fester Schwellwert L für Differenz M-m als Maß für die Streuung)
• d) Die markierten Punkte in den Zwischenergebnisbildern werden dilatiert, d. h. auf 3 x 3 pixel vergrößert. Dies dient der Vorbereitung auf den nächsten Schritt.
• e) Die UND-Verknüpfung sämtlicher Ergebisbilder aus Schritt d) liefert das Endergebnis. Die Schritte d), und e) stellen die Kombination der Filterergebnisse aus Schritt c) dar. Da jedoch aus Gründen unvollkommener Symmetrie nicht alle Filterzweige dasselbe Pixel markieren, ist Schritt d) vor der UND-Verknüpfung erforderlich.
• f) Die Anzahl der markierten Pixel in jedem Filterzweig soll nicht zu groß oder zu klein sein. Um kontrastreiche und kontrastarme Szenen gleichmäßig gut verarbeiten zu können ist eine Regelung erforderlich. Diese Regelung kontrolliert die Anzahl markierter Pixel in jedem Zwischenergebnisbild derart, daß die Anzahl ungefähr konstant bleibt.

The process itself is divided into the following steps:

• a) The incoming image is first passed through a two-dimensional FIR filter to reduce white noise.
• b) The image filtered in this way is processed further by four nonlinear filters which work in parallel. These filters work within a range of e.g. B. 7 x 7 pixels. The filters find the maximum and minimum of the gray values of a group of pixels, which are each located in the corners of the 7 x 7 area (matrix). The special selection of the different pixel groups can be seen in FIG. 1 (hatched triangles).
  To simplify matters, the triangular group of pixels is now called the reference set . This processing step takes into account the symmetry requirements for the detection of small, compact objects. For two-dimensional objects, at least three reference sets and therefore at least three parallel filter branches are required.
• c) In the next step, the results of the maximum and minimum calculations are compared with the current image point (pixel) in the center of the matrix in each filter branch.
  The central pixel is marked as significant and its entry in the intermediate result image of the filter branch takes place if the criteria
  
  $\text{P - M> T (K1)}$
  
  
  
  and
  
  $\text{M - m <L (K2)}$
  
  
  
  are fulfilled.
  Mean:

  P:

  Intensity of the central pixel

  M:

  Maximum of the intensities of the reference set

  m:

  Minimum of the intensities of the reference set

  T:

  dynamic threshold

  L:

  fixed threshold

  The purpose of this processing step is to mark the central pixel if both the pixel is sufficiently lighter than its surroundings (dynamic threshold T) and its surroundings are not too strongly structured (fixed threshold L for difference Mm as a measure of the scatter)
• d) The marked points in the intermediate result images are dilated, ie enlarged to 3 x 3 pixels. This serves as preparation for the next step.
• e) The AND operation of all results from step d) provides the final result. Steps d) and e) represent the combination of the filter results from step c). However, since, for reasons of imperfect symmetry, not all filter branches mark the same pixel, step d) is necessary before the AND operation.
• f) The number of marked pixels in each filter branch should not be too large or too small. In order to process high-contrast and low-contrast scenes equally well, regulation is required. This control controls the number of marked pixels in each intermediate result image such that the number remains approximately constant.

oder

${\text{T}}_{\text{min}}{\text{< T < T}}_{\text{max}}\text{ (K4)}$

erfüllt ist.For this purpose, the threshold value T is incremented or decremented from image to image until the criterion

${\text{N}}_{\text{min}}{\text{<n <N}}_{\text{Max}}\text{(K3)}$

or

${\text{T}}_{\text{min}}{\text{<T <T}}_{\text{Max}}\text{(K4)}$

is satisfied.

n:

Anzahl markierter Pixel in einem Filterzweig

N_min:

untere Schranke

N_max:

obere Schranke

T_min, T_max:

unterer und oberer absoluter Grenzwert für T

N_min, N_max

sind nur abhängig von der Bildgröße.

Mean:

n:

Number of marked pixels in a filter branch

N _min :

lower bound

N _max :

upper bound

T _min , T _max :

lower and upper absolute limit for T

N _min , N _max

are only dependent on the image size.

A circuit arrangement for carrying out the method and further developments of the circuit arrangement are claimed in claim 8 and in dependent claims 9 and 10.

Fig. 1

die Darstellung von vier verschiedenen Referenzmengen zur Erläuterung des Verfahrens,

Fig. 2

eine Schaltungsanordnung zur Durchführung des erfindungsgemäßen Verfahrens in Blockschaltbildform, und

Fig. 3

ein Blockschaltbild des Ergebnisprozessors aus Fig. 2.

In the drawing, an embodiment of the invention is shown and shows

Fig. 1

the presentation of four different reference quantities to explain the process,

Fig. 2

a circuit arrangement for performing the method according to the invention in block diagram form, and

Fig. 3

2 shows a block diagram of the result processor from FIG. 2.

2, a digitized input image supplied by an optoelectronic sensor, not shown in the drawing, is smoothed by a 3 × 3 FIR filter 1 and then via a time delay circuit 2, for example a 7-line delay block, the inputs of four parallel maximum-minimum Filters 3a to 3d fed. These filters extend each delayed image line by a further seven image points by means of registers not shown in the drawing, so that 48 neighboring points of each central image point are available internally (7 x 7 matrix). Using cascaded comparators and multiplexers, also not shown, four different reference sets are formed from the delayed image signals. As shown in FIG. 1, these reference quantities can be arranged at the four corners of a 7 × 7 square matrix.

Using the comparators and multiplexers, each filter 3a to 3d determines the maximum and minimum of the current brightness values from each reference quantity. These four maxima and four minima determined are compared with the central pixel identified by the matrix.

This central point is marked if and only if the two criteria are met. That is, firstly when the current central pixel is brighter than the maximum of the associated reference quantity and secondly when the difference between maximum and minimum of the associated reference quantities is smaller than the predetermined threshold value. The control processors 4a to 4d, which are simple subtractors and comparators, use the pixel intensity values and contain registers for storing thresholds T _a to T _d and barriers N _max , N _min determined in the previous image in order to store the pixels mark and pass on to the result processor 6. From now on, the marked pixels are called set elements.

• daß die Schwellwerte T erhöht werden, wenn die aktuellen Zählergebnisse n größer als eine vorgegebene obere Schranke N_max sind,
• daß die Schwellwerte erniedrigt werden, wenn die aktuellen Zählergebnisse n kleiner als eine vorgegebene untere Schranke N_min sind, und
• daß die Schwellwerte unverändert bleiben, wenn die aktuellen Zählergebnisse innerhalb des von der oberen und unteren vorgegebenen Schranke definierten Bereiches liegen oder gleich der unteren bzw. oberen vorgegebenen Schranke sind.

These quantity elements n are also fed as clocks to the counters for n 5a to 5d downstream of the control processors 4a to 4d in order to set the threshold values for the next images in the control processors. This is done by comparing the current quantity element count, n, with the fixed value of the lower and upper bounds according to criterion K3, which are stored in registers within the four counting blocks 5a to 5d. The threshold values can be regulated in accordance with criterion K3 in such a way that

• that the threshold values T are increased if the current count results n are greater than a predetermined upper limit N _max ,
• that the threshold values are lowered when the current counting results n are less than a predetermined lower bound N _min , and
• that the threshold values remain unchanged if the current count results lie within the range defined by the upper and lower predetermined bounds or are equal to the lower and upper predetermined bounds.

The set elements of all four intermediate result images are dilated and AND-linked in the result processor 6 to a final result set as detected pixels. The coordinate values of the final result set elements are stored in an output memory 12 present in the result processor (cf. FIG. 3).

• aus einem Adressen-Generator 7, dessen Synchron-Eingang mit dem Takteingang kombiniert ist, um die erforderlichen Bildpunktadressen zu erzeugen,
• aus an den Ausgang des Adressen-Generators 7 angeschlossenen Speichern 8a bis 8d, (First-in-First-out-Speicher), die jeweils die Adressen der Mengenelemente speichern, wobei die Mengenelement-Signale als Schreibimpulse verwendet werden,
• aus den Speichern 8a bis 8d nachgeordneten Dilatoren 9a bis 9d zur Vergrößerung der als Mengenelemente markierten aktuellen Bildpunkte, wobei als Dilatoren Look-up-Tabellen vorgesehen sein können, die für jede Adresse eine vorgegebene Anzahl von weiteren Adressen generieren,
• aus den Dilatoren 9a bis 9d nachgeordneten Karten-Speichern 10a bis 10d zur Speicherung der vergrößerten, adressierten Mengenelemente, und
• aus einem an die Ausgänge der Karten-Speicher 10a bis 10d angeschlossenen Und-Glied 11, an dessen Ausgang der Ausgangsspeicher 12 der Auswerteeinheit angeschlossen ist, um die Koordinatenwerte der detektierten Bildpunkte im Ausgangsspeicher 12, z. B. einem First-in-First-out-Speicher, zu speichern.

As can be seen from FIG. 3, the result processor 6

• an address generator 7, the synchronous input of which is combined with the clock input in order to generate the required pixel addresses,
• from memories 8a to 8d (first-in-first-out memory) connected to the output of the address generator 7, each of which stores the addresses of the set elements, the set element signals being used as write pulses,
• from the memories 8a to 8d downstream dilators 9a to 9d for enlarging the current pixels marked as set elements, it being possible to provide look-up tables as dilators which generate a predetermined number of further addresses for each address,
• from the dilators 9a to 9d downstream card memories 10a to 10d for storing the enlarged, addressed quantity elements, and
• from an AND element 11 connected to the outputs of the card memories 10a to 10d, to the output of which the output memory 12 of the evaluation unit is connected in order to determine the coordinate values of the detected pixels in the output memory 12, e.g. B. a first-in-first-out memory.

Each of the pixels of a filter branch marked as a set element is enlarged by including nearest neighborhood points. This process step increases the final probability of detection of the objects because natural scenes are never free from noise and the objects are not do not always have perfect symmetries. For example, each pixel of a filter branch marked as a set element can be enlarged by a total of eight neighborhood points. The eight pixels lying on two lines arranged perpendicular to one another and on the associated diagonals can be selected as the next neighborhood points.

The method according to the invention limits the number of current pixels (objects) marked as set elements to the most important ones, with a required data reduction being achieved for further image processing. The upper and lower bounds, which depend on the image size as a system constant, are advantageously determined and set only once for the system. No other parameters need to be adjusted. The four threshold values of the control processors 4a to 4d can only be increased or decreased between two limit values T _max , T _min according to K4. This avoids extremely high sensitivity if the image is empty and unnecessarily high data reduction if the image is full of interesting objects.

• a) Detektion und Überwachung von kleinen lebenden Organismen mit Symmetrien in biologischer, medizinischer und normaler Umwelt (Testumgebungen).
• b) Detektion von kleinen Defekten in der Produktionsindustrie, z. B. Unreinheiten in Metall, Kristall, Chemiesubstanzen, optischen Komponenten etc.
• c) Überwachung der Anwesenheit oder Abwesenheit von Gas-Blasen-Formationen in chemischen Reaktoren.
• d) Entdeckung und Regelung der An- oder Abwesenheit von Bohrlöchern bei der Herstellung von mechanischen Komponenten (Metall-Teile, PC-Boards etc.)
• e) Entdeckung und Überwachung von natürlichen oder vom Menschen erzeugten Katastrophen lokaler Natur von Erdbeobachtungssatelliten (Waldbrände, Ölfeldbrände etc.)
• f) Detektion von dünnen Brüchen (Rissen) in Material oder kritischen Konstruktionen wie in Kernreaktoren Brücken etc. Die Detektion von Rissen ist eine interessante Anwendung des allgemeinen Verfahrens, da für diesen Fall nämlich einige der Teillösungsmengen der Symmetrieregeln ausgeschlossen werden müssen, wenn die logische Endverknüpfung durchgeführt wird.
• g) Registrierung von Elementarpartikelspuren im Bereich Hochenergiephysik.
• h) Detektion von punktförmigen Objekten in IR-Bildern für die Gebäudeüberwachung.

Possible application examples are listed below, where the input images can come from different sources such as. B. normal video cameras, X-ray photos, infrared images, ultrasound images, optical microscopes, electron microscopes etc.

• a) Detection and monitoring of small living organisms with symmetries in biological, medical and normal environments (test environments).
• b) Detection of small defects in the production industry, e.g. B. impurities in metal, crystal, chemical substances, optical components etc.
• c) Monitoring the presence or absence of gas-bubble formations in chemical reactors.
• d) Discovery and control of the presence or absence of boreholes in the manufacture of mechanical components (metal parts, PC boards etc.)
• e) Discovery and monitoring of natural or man-made disasters of local nature from earth observation satellites (forest fires, oil field fires etc.)
• f) Detection of thin breaks (cracks) in material or critical constructions such as in nuclear reactors, bridges etc. The detection of cracks is an interesting application of the general method, because in this case some of the partial solution sets of the rules of symmetry have to be excluded if the logical end linkage is carried out.
• g) Registration of elementary particle traces in the field of high energy physics.
• h) Detection of punctiform objects in IR images for building surveillance.

Claims (10)

Method for the detection of small objects in a natural environment using an electro-optical sensor which scans the environment with a downstream evaluation unit and, if applicable, an image display device, characterized by the following method steps: a) a digital image supplied by the sensor is filtered by at least three nonlinear filter branches operating in parallel, starting with maximum-minimum filters (3a to 3d) such that each filter (3a to 3d) has the maximum and minimum of the brightness values in each case one of at least three different reference quantities is determined, the number and shape of the groups of pixels being selected such that they correspond to the symmetry properties of the objects to be detected, b) the central pixel is marked according to the two criteria K1, K2 if, firstly, this pixel is brighter than the maximum of the brightness values of the associated reference quantity and secondly if the scatter of the associated reference quantity is smaller than a predetermined fixed threshold value L, taking as a measure the difference between maximum and minimum is used for the scatter, c) the number of quantity elements (intermediate results) is counted for a complete image, and the current count results are compared with the fixed bounds N _max , N _min and used to regulate the threshold values T in accordance with criterion K3, K4, that the threshold value T is increased when the current count results n are greater than a predetermined upper limit N _max , - That the threshold value T is lowered when the current counting results n are less than a predetermined lower limit N _min , and that the threshold value T remains unchanged if the current count results lie within the range defined by the upper and lower predetermined bounds or are equal to the lower or upper predetermined bounds, and, d) each pixel marked as a set element is dilated by 1 pixel, ie after dilation each marked pixel appears enlarged by its neighboring pixels, and the enlarged, ie dilated set elements are AND-linked to a final result, and the coordinate values of all points thus detected are stored in an output memory (12) of the evaluation unit. Method according to claim 1, characterized in that the noise of the digitized input image is smoothed by a pre-filter process. Method according to Claim 1, characterized in that each pixel marked as a set element is enlarged by a total of eight neighborhood points. Method according to Claim 3, characterized in that the eight pixels lying on two lines arranged perpendicular to one another and on the associated diagonals are selected as the next neighborhood points. Method according to one of Claims 1 to 4, characterized in that a total of four reference quantities (see FIG. 1) are used, the maxima and minima of which are determined by four nonlinear maximum-minimum filters in order to assign these values in the following control processors for marking purposes use. Method according to Claim 5, characterized in that four groups of pixels arranged at the corners of a square image area are used in accordance with the illustration shown in FIG. 1. Method according to one of Claims 1 to 6, characterized in that the variance of the image points is used as the measure for the scatter instead of the difference between maximum and minimum. Circuit arrangement for carrying out the method according to one of claims 1 to 7, characterized in that an optoelectronic sensor is followed by a number of parallel-arranged maximum-minimum filters (3a to 3d) corresponding to the examined number of reference quantities, which filters the maxima and minima Determine the associated reference quantities and which expand the quantity elements marked as current pixels by dilation that the outputs of the maximum-minimum filters (3a to 3b) to the inputs of a downstream control processor (4a to 4d) to determine the number of quantity elements in each Group of pixels (intermediate results) are connected that the outputs of the control processors (4a to 4d) on the one hand to a counter (5a to 5d) for counting the number of quantity elements for the purpose of regulating the threshold values in the control processors (4a to 4d) and on the other the inputs of a result processor (6) for AND gating ng of the quantity elements are connected to a final result quantity, and that an output memory (12) for storing the coordinate values of all final result elements is arranged in the result processor (6). Circuit arrangement according to Claim 8, characterized in that a pre-filter (1) and a time delay circuit (2) are arranged in front of the maximum-minimum filters (3a to 3d). Circuit arrangement according to claim 8, characterized in that the result processor (6) consists: - from an address generator (7), - from memories (8a to 8d) connected to the output of the address generator (7), the number of which corresponds to the number of reference quantities and which each store the address of the quantity elements, - from the memories (8a to 8d) downstream dilators (9a to 9d) to enlarge the current pixels marked as set elements, - from the dilators (9a to 9d) downstream card memories (10a to 10d) for storing the enlarged, addressed set elements, and - From an AND element (11) connected to the outputs of the card memories (10a to 10d), to the output of which the output memory (12) of the evaluation unit is connected.

Images