DE102006060612B4 - Method for monitoring target objects and multispectral camera - Google Patents

Abstract

Method for evaluating images in a multispectral camera, digital multispectral image data being transmitted from an image sensor unit of the multispectral camera to a processing unit of the multispectral camera and a color histogram being generated in the HSV color space for each pixel of a digital multispectral image generated by the image sensor unit,
characterized in that
the brightness values of the individual pixels are scaled and a database-aided determination and marking of the transition points between a target object and the background is performed, wherein
calculating a set of difference values of the color and texture of the pixel to be examined and the mean color and texture of a number of pixels relative to the pixel to be examined in the same row / column respectively before / above and behind / directly adjacent pixels, the color being marked is characterized by the averages of hue, saturation, and brightness, and the texture is characterized by the standard deviations of hue, chroma, and brightness, and where
for the determination of target objects row-wise / column-wise two arranged in a row / column by a predefinable ...

Description

The The invention relates to a method for monitoring target objects according to the characteristics of Claim 1 and a multi-spectral camera according to the features of the claim 17th

in the Field of military clarification, clearing up of something For example, using sensors scenarios, especially in a terrain recorded after the presence of facilities, vehicles, Infrastructure features and the like should be investigated. This will be done by means of surveillance cameras a big Number of pictures available which are evaluated under predetermined time conditions are. The objects to be recognized have any dimensions and can one this characterizing structure with a more or less large complexity exhibit.

With known systems is a soil reconnaissance and detection and persistent Ground monitoring greater Area (eg 100 km to 1000 km border area with a few kilometers Depth) from the air of passenger, carrier, and vehicle targets possible. These are usually Used multispectral stereo cameras mounted on aircraft record the data in flight. The data will be sent to you after the flight Soil evaluated, which generally takes hours to days until the reconnaissance and discovery results are available. An immediate monitoring is thus not possible. on the other hand are video cameras, partly also equipped with detection aids, for permanent monitoring smaller areas used from the air, with the target search and Assessment to the observer stays, and the data easy for latter Postprocessing will be recorded. For monitoring large areas This method is not suitable because of the extremely high cost for this.

Out DE 10 2005 009 626 A1 is a camera known for tracking objects. In this camera, target objects are identified by comparing the captured image with a database. A similar method for target recognition is off DE 199 55 919 C1 known. These methods have the disadvantage that the recognition of targets is very slow.

Out XU Richard Y.D. et al: "Robus real-time tracking of non-rigid objects "; ACM; 2004; Proceedings of the Pan-Sydney area workshop o Visual information processing; Conference Proceeding Series, Vol. 100, pages 95-98; ISBN-ISSN: 1445-1336, 1-920682-18-X, a method is known which of selected by a user and can track moving objects in video sequences.

BALA Jerzy, et al. "Applications of Distributed Mining Techniques for Knowledge Discovery in Dispersed Sensory Data "; 7th Joint Conference on Information Sciences - JCIS 2003, Cary, NC, USA, September 26-30, 2003, Data Mining Session describes a procedure which consists of Can recognize images of target objects. These are different types from masks over the pictures taken.

YOU Peijun, et al .: "Study to content-based remote sensing image retrieval, "IEEE, 25-29 July 2005, Proceedings of the Geoscience and Remote Sensing Symposium 2005. (IGARSS '05), Volume 2, page (s): 707-710; ISBN: 0-7803-9050-4 describes a method in which remote sensing via masks database support Color and texture data are compared.

It Object of the invention to provide a method with which a surveillance of target objects during of the overflight of an area to be observed is. Another object is to provide a multispectral camera, which is able to fly over one great Real-time target Object with low contrast to the background to discover and classify.

proposed becomes a multispectral camera for monitoring target objects with a digital image sensor unit for taking pictures and for generating digital multispectral image data and a processing unit for processing the data transferred from the image sensor unit to the processing unit Multispectral image data in which the processing unit has a calculation unit for Calculation of a color histogram in the HSV color space for each Image pixels of a digital image and a classifier for database-supported recognition of boundaries between target objects and target background.

The Processing of image data is done directly in the camera, so that not the entire full-resolution data stream to an external processing unit, z. B. transmitted a ground station must become. Detected target objects are transmitted to the ground station.

In In a further development of the invention, the multispectral camera is a Multispectral stereo camera, and a calculation unit for determining the height of a target object from two is present with a stereo angle recorded multispectral images.

If two images with a usable stereo angle (angle of the image axes of the two images) are present from the same target area, one of the stationary target objects found can be used Altitude measurement be made.

The The problem underlying the invention is also due to a method according to the characteristics of claim 1.

In order to is an image acquisition by a narrow-band multispectral (blue, Green red, Near-infrared) aerial camera and for automatic real-time image analysis, Target object classification and detection, and if necessary Create a target track list for target tracking aboard Flights possible. The automatic image analysis is based on the target classification in front of a specific associated Background taking advantage of the special information content of stereo multispectral recordings. This will be low contrast targets from a background with many potential false targets with very low false alarm rate and high reliability discovered, classified and recognized, and moving goals are considered Detected moving targets and measured their velocity vector, and of the found goals, a target object track list is created with target signature, position, and velocity vector for further target tracking. The evaluation in flight also allows a significant data reduction the to be transmitted to the ground station Dates.

Usually A multispectral camera generates images with image data in RGB format. The image data is converted from the RGB format into the HSV (hue, color saturation, Brightness) format, in which all further calculations carried out become.

The The invention will be explained in more detail with reference to figures. Show it:

1a the display of a brightness and modulation increase in bright image areas,

1b the display of a brightness and modulation increase in dark picture parts,

2 the representation of a classifier for the discovery of essentially monochromatic target objects,

3 the presentation of a classifier for the discovery of camouflaged targets.

In a development of the invention, the brightness distribution in the camera image is improved in the dark and bright image parts in the direction of a uniform modulation depth ( 1a . 1b ), so that bright and dark parts of the picture can be viewed equally well. This is done in a suitable manner in that in each case for a predeterminable section of a line, for. 30 pixels, the local average MW of the brightness and the standard deviation ST of the brightness are calculated and then the brightness value is scaled by the quotient of the mean plus the standard deviation divided by the mean. This results in a lightening of dark image areas and in an intensification of the resulting colors, by which smaller color differences are made visible.

Discovery of monochrome targets

version 1

For recognition of essentially monochromatic target objects, the improved images are examined by a color classifier (1st color classifier), line by line or column by column ( 2 ). In this case, it is checked whether at the observed pixel BP in an image to be examined, with the reference character 8th the line / column of the image to be examined is specified, there is a transition from a background color and a background texture to a target color and target object texture respectively.

For this be in a first embodiment of the color classifier for the sought Target objects first a target object reference color table of the allowed target colors and Target object textures created. Likewise, for all legal backgrounds, before which target objects can be searched, a background reference color table created.

Under the term classifier window 2 is understood below the image detail from the camera image.

The color of a target or background is characterized by averages of hue, chroma, and brightness over a classifier window 2 , The texture of a target or background is characterized by the standard deviations of hue, color saturation, brightness, and brightness (increase or decrease). About the texture of the background z. As road, meadow, or passable surfaces are detected. About the texture of the target can be z. B. recognize paint color of vehicles.

It is useful in an area 4 in front and one area 5 after each pixel considered BP in the direction 1 one row / column each for a picture window 2 (Classifier window), which is adjusted in size to the size of the searched target objects, examines whether the predefinable conditions for the searched background target object transition are met, eg. For example, background color and target color, or background texture and target texture have a specifiable difference. If This will be the background-to-destination transition in the picture 8th entered. The same process is then conveniently applied to target-to-background transitions and also into the image 8th entered.

at complex coloring arise in this process step but many false positives The false alarms can expediently greatly reduced Be that only together Background-to-destination and then directly following target-to-background transitions, the not separated by further transitions are as valid Couple scored and for the further calculation will continue to be used. This color classifier is applicable to targets that have sufficient color or texture contrast over the Have background.

The Location of the transition determined by the classifier is correct due to the statistical nature of the process, which only averages and standard deviations considered, not exactly coincident with the visually perceptible border of the target object, but usually lies before or behind or above or below. For the exact definition of the position and outlines of the target object will be useful along the Row / Column 15 pixels before and after the transition found at a distance of z. B. 3 pixels a classification of the found Color (HSV value) according to permissible, predefined Target colors and background colors of a fixed color palette made. This happens because each of the normalized sum of the square squares the HSV values from the individual colors of the color palette to the examined Color calculated, and the inspected color of the target palette color assigned with the smallest square of the distance and thus classified and the position of the examined positively-classified pixel as a new position of the considered background-target transition is treated. This ensures that in most cases the Target object mark to lie within the outline of the object comes and a permissible one Target reference color is assigned, and an actual Position within the object for further investigation disposal stands.

Variant 2

In a second embodiment of the color classifier (2nd color classifier) The target and background reference color tables become special for the straight considered row / column respectively from a reference outline image of a specific searched target object type (eg car, truck, carrier, Group of people) in a specific reference coloration sought or reference texture created.

In the further process steps, the second Farbklassifikator works like the 1st color classifier for possible targets in the too find and locate the examining image. Deviating from the 1. Color classifier becomes the second color classifier in one go for background-to-destination transitions, destination-to-destination transitions, and destination-to-background transitions simultaneously searched and, respectively, the position of the first possible one encountered Target in line / column direction held, the found Position indication due to the statistical nature of the classification process usually next to the exact middle of the actual Target object is located.

A preferably accurate measurement of the position of the center of the target along the Line is now achieved by starting from the starting position respectively a classifier window width before and after this position of the Section of measurement line / column within the classifier window in the reference outline image of the searched target object type in the searched Target color is selected is and the distance sum over all HSV color values (hue, color saturation, brightness) and standard deviations (of hue, color saturation, Brightness) is added up. Then the total distance totals become for all compared to the tested positions and the position with the smallest Distance total scored as the most likely target position. Hereby a complete Correlation of the reference object strip to the found target strip takes place, the target search process of the 2nd color classifier is much more selective and accurate than the 1st color classifier. The 2nd color classifier but also requires a higher one Computational effort.

Discovery of camouflaged targets

version 1

For recognition of camouflaged target objects, the 1st color classifier is modified, also referred to below as the 1st camouflage color classifier. This camouflage color classifier works with two two-dimensional windows arranged one behind the other 2 . 3 , which along a row / column in the direction 1 be pulled over the camera image. On the leading scan window 2 (Classifier window) is a reference pattern 4 camouflage pattern containing all the essential elements of the pattern. The dimension of the sampling window 2 in both directions is suitably three to four times the mean spot size.

A stain size is usually understood to mean the size of an irregularly shaped surface consisting of a camouflage pattern. Camouflage pattern are generally composed of irregularly shaped surfaces, which in the area on the size of the vegetation elements, before which as a background, the object to be camouflaged should no longer be distinguishable, and are matched in their color to the color of the background. Thus, the sampling window includes 2 all the essential elements of the pattern.

For every single pixel 6 of the reference pattern 4 in the sampling window 2 is against every pixel 6b of the target object window to be tested 7 (Extract from the camera image) in each case the least square of the HSV values is calculated and this reference point is assigned the smallest value as the distance. The sum of all distances of all reference points is then the total distance of this object window position from the target object pattern.

On the following sampling window 3 is the reference pattern 5 the background color pattern having the shortest distance to the target object pattern found. This pattern will then be used as the background reference pattern closest to the actual background. The dimension of the reference pattern window 3 is suitably three to six times the mean spot size in both directions. Thus, the reference pattern window includes 5 all the essential elements of the background pattern.

For every single pixel 6 of the reference pattern 5 is against every pixel 6b of the background window to be tested 7a of the camera image is calculated in each case the distance square sum of the HSV values and the smallest value is assigned to this reference point as the distance. The sum of all distances of all reference points is then the total distance of this object window position from the background pattern. The target object window position along the search path in which the distance of the target reference pattern to the target window is smaller than the distance of the background reference pattern to the target window and the distance of the target reference pattern to the background window is greater than the distance of the background reference pattern to the background window, is considered to be the most probable position of the camouflaged object , In addition, the distance of the target reference pattern to the target window must be smaller than a previously determined distance barrier, so that the object found at this point can be considered a valid positively classified target object.

These Examination then becomes longitudinal of the search path for all target object reference patterns to be searched are executed and thus looking for these patterns.

The absolute location of the target object can not be more accurate than that half window size determined be, the relative position z. B. in two consecutive pictures, in short Distance were recorded, but is accurate with some pixels enough for one Stereo height over background Measuring the found target with an accuracy of a few pixel widths.

Variant 2

to Detection of specific searched target object types (eg cars, Truck, pack animal, group of people, fresh mounds, hut) becomes the 2nd color classifier modified, also referred to as the 2nd Tarnfarbenklassifikator. For this purpose, a classifier is used, with two consecutive arranged longitudinally the row / column drawn, two-dimensional windows works, its dimensions along and across the row / column equal to the classifier window width of the 2nd color classifier. Located on the leading window (classifier window) a reference pattern of the sought camouflage pattern, the same as the 2. Color classifier described in advance in a reference table is created to characterize the target object, and that's big enough is to all the essential elements of the pattern and the detailed Contain outlines of the object part captured by the window. This is at three to four times the mean spot size in both Given directions, that is The object can be with three adjacent windows in its characterizing outlines are detected.

Then, for each individual pixel of the searched reference pattern, the least square sum of the HSV values is calculated against each pixel of the tested target object window, and the smallest value is assigned to this reference point as the distance. The sum of all distances of all reference points is then the total distance of this object window position from the target object pattern. Starting from the object position measured with the 2nd color classifier, the target object window will now be in smaller subfields 9 at a distance of one third of the window size around the starting position in all 8 positions in which the smaller windows group around the middle window of the 3 × 3 array, the distance sum is determined as described above. The position of the test grouping with the absolute minimum total distance is then evaluated as the most probable true position of the image object found and recorded for later use.

The classification of the target object can be further refined in Variant 1 and Variant 2. To do this, the target object scan window becomes 2 in z. B. split nine sub-windows in a 3 × 3 arrangement. For these subwindows, the distance totals to the corresponding subwindow in the classifier window are calculated again. This process is now, with each by a window width of the sub-windows along the circumference offset sub-window repeated until all eight possible rotational positions are recorded. During rotation, the central window forms the axis of rotation z, around which the other 8 panes rotate. In each rotational position, these 8 sub-windows 9 each shifted by one position in or counterclockwise.

The rotational position with the smallest distance total is considered the most probable true rotational position of the target object found and recorded. This process is repeated for each of the object shape types to be searched for each of the 4 possible orientations, ie for 4 object shape types z. As car, truck, Tragtier, group of people in 4 orientations must the process 16 to be repeated. The mentioned 4 orientations a, b, c, d result in a window in a 3 × 3 arrangement of partial windows from an arrangement as shown in FIG 3 is shown.

Of the Object shape type with the smallest total distance is considered the most probable Object shape type evaluated and recorded, along with the object shape type (eg car, truck, carrier, group of persons) of the reference object used, the color pattern type (monochrome, camouflage pattern), and the middle Object color in its six characteristic values (averages and Standard deviations), and orientation in space through the measured rotational position.

Determination of signatures

The in the described variants of color classifiers and camouflage color classifiers described method can on the one hand, nationwide Used to search for objects with known reference patterns become.

on the other hand can the methods are also applied to already recognized target objects, to classify them according to known target object reference patterns. With this found classification, the actually found Target object patterns are recorded around the found target position, a distinctive signature of this specific target to obtain. With the help of this signature, the target object in a later Aerial photograph of the same area in a meanwhile done locomotion across from the first shot, and the object in question be classified as a moving object, as well as direction of movement and Speed of the target object to be measured against the background. These are all the target objects found in the first stereo recording and with their signature were recorded in the (by a maximum half image width shifted) second stereo recording on the same Searched by correlation and determined which objects were recovered, which are stationary, and which objects are no longer in the old place, so have to have moved. These presumed moving objects are in the second stereo recording in a radius, reached the object sought between the two recording times could have (using realistic equations of motion) Hand of the target object list for searched the second stereo recording (by signature comparison with agreement). If the object is found, then from the location difference and the time interval the speed and the direction of movement of the object. These results allow it, the target object lists of all recordings in an overall object record and from this by tracking the movement the moving objects a total track record of the observed Area by repeated overflight over the area also temporally continued for any length of time and can be sustained. If the track record over a long time can be maintained so creates a complete directory the objects with their detailed signature, as well as a situation picture and a motion profile with respect to the found target objects in the observed area.

Automatic target recognition

The Found actual Target signatures and the found target classification can be found in a post-processing step on the ground to the on-board evaluation be used in the air, the reference target object pattern to improve that z. B. by means of a Gradientensuchverfahrens the reference target object pattern is changed until the total distance sum of the reference target object pattern to all positively classified ones found actual Target object patterns reached a minimum value. The maximum value of observed while still positively classified distances to new reference pattern is recorded as a distance barrier, the for a successful classification must not be exceeded. This calculated new reference pattern is then generalized as follows Target object reference pattern with the determined distance barrier for this Class used. With that you can the target object reference patterns through experience from the evaluation of the indeed found target objects are improved and make the system in capable of learning in this sense.

A Another application is the existing reference target object pattern by inserting new pattern to complete, when it is detected that the target objects found in two Groups with smaller permissible Split gap barriers, which then as Klassifikatoreinstellung Classify more selectively than the original ones.

The found targets, which still have a share of false targets can contain in the further with a method for automatic target recognition (ATR) over the object shape is examined according to learned reference forms. This ATR procedure can usually only use monochrome brightness images (gray value pattern).

In In a further development of the invention, the ATR method is modified, to multicolored, z. B. three or four color axes comprehensive multispectral color spaces, and to recognize complex color patterns composed of these.

This is done appropriately, that instead of the brightness value the normalized distance sum over all Spectral colors of the color space from a suitably selected target reference color inverted (that is, subtracting the one normalized value of one is) is used as the brightness value. This makes the ATR process complex Recognize and separate color patterns that are unique in their brightness values do not differ and thus according to the brightness values alone are not separable.

height determination

Lie from the same target area two recordings with a usable stereo angle (Angle of the image axes of the two images) before, so can in one Development of the invention of the found stationary targets made a height measurement when one of the calculated target diameters in the direction of expected stereo shift of the examined object is located. This will be an area along the row / column in the first image that completely covers the object, Pixel for Pixel above that pushed the second picture, and for each position the sum of Distance squares in HSV for the window in the first picture and in the second picture is calculated. The position with the least distance is then the most likely position of the object in the second picture. Taking into account the stereo angle it can be the measured height of the object over be calculated against the background.

This can when using a line scanner camera or a non-stabilized Full-frame camera already with the unrectified raw data accomplished become. This is achieved appropriately, that a stereo height over background Surveying the target objects of interest directly in flight can be achieved after admission with on-board means, and not only after the flight on the ground after the time-consuming rectification calculation. This allows the footage to be targeted immediately after shooting be scanned, which have a measured height, and you can thus false targets, not enough height exclude from further processing. This allows the stereo height measurement of target objects also in ongoing surveillance flights to Immediate image evaluation will be used, reflecting the effectiveness the surveillance can significantly improve without overly increasing the effort, z. B. can do so camouflaged targets are more easily recognized or flat images of Objects of real high-heeled Objects are distinguished.

Claims (14)

A method for evaluating images in a multispectral camera, digital multispectral image data being transmitted from an image sensor unit of the multispectral camera to a processing unit of the multispectral camera and a color histogram generated in the HSV color space for each pixel of a digital multispectral image generated by the image sensor unit, characterized in that the brightness values of a pixel-based determination and marking of the transition points between a target object and the background is performed, wherein a set of difference values of the color and texture of the pixel to be examined and the mean color and texture of a number of relative to the pixel to be examined in the same line / column respectively in front of / above and behind / below directly adjacent pixels are calculated, the color being characterized by the average values of hue, color saturation and Brightness and the texture is characterized by the standard deviations of hue, color saturation and brightness and wherein for the determination of target objects row / column by two arranged in a row / column by a predetermined number of pixels spaced two-dimensional sampling window ( 2 . 3 ) via the digital multispectral image ( 8th ), whereby the target object sampling window ( 2 ) a prescribable reference image ( 4 ) of a color pattern of a target object and the background scan window ( 3 ) a definable reference pattern ( 5 ) of a color pattern of a background and a set of equidistant square sums of the HSV values of each pixel of the target / background scan window (FIG. 2 . 3 ) to each pixel of a size target / background window ( 7 . 7a ) on the digital multispectral image ( 8th ), wherein the pixel of the target / background scan window ( 2 . 3 ) is assigned the smallest value of the set of distance sums sums and the total distance of the target object / background window ( 7 . 7a ) to the target / background scan window ( 2 . 3 ) is calculated from the sum of the individual distances of the individual pixels, wherein the Location of the target object / background window ( 7 . 7a ) on the digital multispectral image ( 8th ) is defined as the position of a target object when the distance of the target object scanning window ( 2 ) to the target object window ( 7 ) is smaller than the distance of the background sampling window ( 3 ) to the target object window ( 7 ) and the distance of the target scan window ( 2 ) to the background window ( 7a ) is greater than the distance of the background sampling window ( 3 ) to the background window ( 7a ) and the distance of the target scan window ( 2 ) to the target object window ( 7 ) is less than a predefinable threshold. Method according to claim 1, characterized in that the target object / background scanning windows ( 2 . 3 ) and the associated color patterns ( 4 . 5 ) of the target object and the background are subdivided into a predeterminable number of subwindows. Method according to claim 2, characterized in that the orientation (a, b, c, d) of a target object is determined by the rotation of the target object window divided into a plurality of partial windows ( 2 ) with the associated color pattern ( 4 ) about a rotation axis z Method according to claim 1, characterized in that that the step determination of target objects is a classification contains the transition pixels, where along the row / column a predefinable number of to be examined Pixels before / about and after / under a detected transition pixel a classification of the color of the transition pixel by color a predeterminable color set of target and background transitions. Method according to claim 4, characterized in that that the step classification of transition pixels the calculation the normalized distance squares of the HSV values of the individual colors of the predeterminable color set to the HSV value of each to be examined Contains pixel wherein each pixel to be examined in each case the lowest normalized Distance square is assigned and wherein the pixel as new transition pixel is marked whose normalized square distance is the lowest. Method according to one of claims 1 to 5, characterized by means of two multispectral images recorded at a predeterminable stereo angle the height a target object is determined without a prior rectification to take the picture. Method according to one of claims 1 to 6, characterized by means of two multispectral images recorded at a predeterminable stereo angle the speed and the direction of movement of a target object is calculated without first rectifying the image. Method according to Claim 6, characterized that the step of height determination of a target object contains the step that an area in the first Multispectral image, which completely includes the target object, along the Line / column in the first multispectral image over the second multispectral image pixel for picture point pushed and for each position taking full advantage of the spectral contrast the Sum of the distance squares between the HSV values in the first multispectral image is calculated to the HSV values in the second multispectral image, where the least squares position is the most likely Position of the target object in the second multispectral image assumed becomes. Method according to one of the preceding claims 1-8, characterized characterized in that each detected target object with a method for automatic target recognition via the object shape is examined for learned reference patterns. Method according to claim 9, characterized in that that for each pixel of the determined target object, the normalized distance sum to a predeterminable multidimensional and multispectral color space calculates this distance sum from a predefinable target reference color subtracted and this result as a brightness value for the pixel is used. Method according to claim 9 or 10, characterized that the method for automatic target recognition by means of a neural network performed becomes. Multispectral camera for performing any of the foregoing Procedure for monitoring of target objects with a digital image sensor unit for recording of images and for generating digital multispectral image data and a processing unit for processing the image sensor unit passed to the processing unit Multispectral image data, characterized in that the processing unit a calculation unit for calculating a color histogram in HSV color space for each image pixel of a digital image and a classifier for database-driven Detection of boundaries between target objects and target background includes. A multi-spectral camera according to claim 12, characterized in that for database-aided recognition of boundaries between target objects and target background, a target reference color table and a background reference color table is available. Multispectral camera according to claim 12, characterized in that that the multispectral camera is a multispectral stereo camera, and a calculation unit for determining the height of a target object from two present with a stereo angle recorded multispectral images is.