DE102023207319B3 - Computer-implemented method and system for determining an object in a recording - Google Patents

Abstract

The invention relates to a computer-implemented method (100) for determining an object in a recording, comprising the steps of: providing (110) a plurality of image files characterizing a plurality of recordings (A) of a predetermined area taken one after the other to an image processing module (300); providing (120), by means of the image processing module (300), histogram information to a control device (400); generating (130), by means of the control device (400), a time-discrete signal based on the histogram information of the recordings (A) taken one after the other; Determining (140), by means of a first trained machine learning algorithm of the control unit (400), a presence of the object in at least one of the recordings (A) based on the time-discrete signal, wherein the first machine learning algorithm is designed to determine the presence of the object based on a temporal change of a gradient of a tone value of at least one histogram of the histogram information.

Description

The invention relates to a computer-implemented method and a system for determining an object in a recording.

Determining and/or recognizing an object in a recording plays an important role in various areas of application, such as driving and working tasks in the industrial sector. For example, objects must be recognized in a field so that they are not driven over. It does not necessarily matter what type of object it is, the important thing is that it is not driven over.

One possible application would therefore be an emergency stop function that stops the vehicle when an object appears in a camera image that was not previously in the image area or recording. This type of object recognition can also be applied to work tasks that are only carried out when a certain object is in the image area or the work task is then aborted. This could be, for example, human-machine collaboration.

One example is the application of colored markings on an industrial floor. As soon as these are recognized in the image area, a vehicle carries out a defined action (e.g. emergency stop or reversal or speed reduction). The color can be coded accordingly (e.g. red for emergency stop, yellow for speed reduction).

Histograms are well known in the field of image processing. A histogram in the field of image processing shows the statistical frequency of the tonal values of an image in the form of a (discrete) bar diagram. It is therefore also called a tone curve. The tonal values (color/brightness values) are on the horizontal axis. The height of a bar indicates how often this tonal value occurs in the image. In the simplest case of a grayscale image, it reflects the number of pixels of the colors black (on the left edge) to white (on the right edge). A histogram can be used to estimate the tonal value distribution and thus the correct exposure (under/overexposure), the contrast and other visual attributes of the source image. There is no way to reconstruct the original image from a histogram, since every different arrangement of pixel values has exactly the same histogram. Histograms can also be created for color images. One possibility is to create a separate histogram for each color channel.

The general term point operators covers various frequently used methods of digital image processing such as tone value correction, contrast enhancement and brightness correction. They usually represent the first step in preprocessing to increase the image contrast or to compensate for exposure errors when taking the image. Segmentation of an image is often achieved by applying a global threshold method. After segmentation, the objects found are often classified, i.e. "sorted" into individual classes. Pseudo-coloring or false color representation methods are commonly used to visualize the results of segmentation or classification for humans. Segmentation is a sub-area of digital image processing and computer vision. The creation of content-related regions by combining neighboring pixels or voxels according to a certain homogeneity criterion is called segmentation.

Global thresholding methods are often used in the field of image segmentation. Each pixel of a grayscale image is assigned to one of two classes based on its gray value. In this way, for example, the foreground of the image is often separated from the background or objects of different brightness are distinguished. However, such a separation can only work optimally if a bimodal histogram is present, i.e. if two local maxima exist in the histogram.

In addition, object recognition in an image (i.e. the recognition of objects that differ from the background in terms of color/intensity of color) is very difficult to perform on a standard control unit or a control unit with low computing power. Computationally intensive "machine learning" image processes are used for reliable segmentation. However, these processes can only segment objects that they already know. For example, training such a network in a field with different vegetation levels for plants is very difficult.

The state of the art, for example, US 5 969 772 A , Video analysis method and a video analysis device for analyzing a sequence of video data or images to detect several moving objects. US 6 445 818 B1 relates to a color image processing method, in particular a method for designating a local representative color value and for automatically determining an optimal image search algorithm on a color image. EN 10 2004 050 375 A1 concerns a method for detecting and classifying traffic-relevant objects based on color images of the environment of a motor vehicle. EN 10 2014 112 914 A1 relates to a method for segmenting an image of the surroundings of the motor vehicle provided by a camera of a motor vehicle. The invention also relates to a driver assistance system for a motor vehicle and to a motor vehicle with such a driver assistance system. US 10 410 084 B2 generally concerns the detection of anomalies in images.

It is an object of the present invention to provide a computer-implemented method and a system which solve and/or at least improve one or more of the aforementioned disadvantages. In particular, it is an object of the invention to enable the presence of an object in a receptacle in a control device to be detected with little computational effort.

The object is achieved according to a first aspect by a computer-implemented method for determining an object in a recording. The method comprises providing a plurality of image files characterizing a plurality of recordings of a predetermined area taken one after the other to an image processing module. The method further comprises providing, by means of the image processing module, histogram information to a control device. The provision of the histogram information comprises, for two or more recordings of the recordings, dividing the recording into a predetermined plurality of sub-segments. The provision of the histogram information can comprise the division and the subsequent steps for each of the recordings. The provision of the histogram information further comprises determining a histogram for predetermined color channels of each sub-segment of the plurality of sub-segments, wherein a histogram characterizes tonal values and a number of tonal values of the color channel of the sub-segment. The histogram information characterizes the determined histograms. The method further comprises generating, by means of the control unit, a time-discrete signal based on the histogram information of the images taken one after the other. The method further comprises determining, by means of a first trained machine learning algorithm of the control unit, a presence of the object in at least one of the images based on the time-discrete signal. The first machine learning algorithm is designed to determine the presence of the object based on a temporal change in a gradient of a tone value of at least one histogram of the histogram information.

The aim can be to recognise an object by a significant occurrence of new/different colour channels. It can be irrelevant what kind of object it is. It is primarily just crucial to recognise that an object is in the field of view of a camera image or recording. In order to reduce the computational effort required for object recognition on a control unit, the above-mentioned method is described, which describes at least one image processing module (such as an Image Signal Processor (ISP)) and the function on the series control unit as components. With the proposed solution, the presence of the object can also be recognised in a relatively homogeneous image. For example, the presence of a stone can be recognised when driving across a field.

The method may be used for use in assisted and/or autonomous driving of a vehicle and/or work of off-highway machines.

Providing the plurality of image files may include recording the plurality of image files by means of a camera unit and sending the plurality of image files to the image processing unit by means of the camera unit. The image processing unit may be included in the camera unit or be another separate unit.

The image files of the multitude of image files can be individual frames of a video recording.

The images can be two-dimensional images.

The subsegments of the plurality of subsegments can be of equal size.

The tonal values can be converted into classes to provide an improved structure of information for computer processes.

In the first step, the image files can include the respective raw images from the camera unit and be divided into equally sized sub-segments, whereby the sub-segments can be variable in both their size and their number (number of sub-segments per image file). This can be done, for example, via the image processing module, which is installed either directly on the camera unit or in a downstream unit.

In the next step, the histogram for each of these sub-segments can be determined on the image processing module for each of these sub-segments. The goal can be to determine the statistical frequency of the tonal values for each existing color channel for each existing sub-segment. These histograms can be calculated for each sub-segment separately to the series control unit via a communication unit. The communication unit and the protocol used can be of any design (SPI, LAN, CAN, LIN, FlexRay).

In the next step, the histograms of the sub-segments can be interpreted via the series control unit. The histogram per color channel can consist of tone value (x-value (e.g. 0-255 in the RGB color space, but not limited to this)) and frequency of this tone value (y-value). The tone values can be defined as classes or converted into such, so that (for the RGB example 0-255) 256 classes exist for each color. Consequently, 256 measured values per color channel per sub-segment can be obtained in each sampling period. These measured values per color channel per sub-segment can then be combined to form the time-discrete signal over time. These can be seen as input features for the first machine learning algorithm (for 3 color channels this would be: 256 * 3 * "number of sub-segments").

One goal can be to use the first machine learning algorithm to map or detect anomalies (for example, large deviations in intensity caused by an object appearing) over time. A neural network can be trained in advance with a labeled data set that includes the relevant objects as labeled information. The neural network can only receive the information whether the object is in the image or recording or not. Based on the temporal change in the gradients of the color channels, the neural network can learn to assign the appearance of an object to this change. This training can be carried out on a computer with high computing power. The neural network can be the trained first machine learning algorithm.

The control unit can comprise a memory for storing the first and/or a subsequently mentioned second machine learning algorithm. The control unit can have a computing power that is lower compared to a computing power of the image processing module. In particular, it can be a so-called embedded control unit, for example an engine control unit, a transmission control unit or another control unit for a subfunction in a system, e.g. a vehicle. An embedded control unit is characterized by a lower computing power compared to a high-performance computer (HPC).

The histogram information may further comprise position information that characterizes respective positions of the plurality of sub-segments in the plurality of image files. The method may further comprise, based on the position information, determining a position of the object in the at least one sub-segment comprising the histogram with the temporally changed gradient.

The method may further comprise determining a direction of movement of the object based on a temporal progression of the position of the object in the sub-segment. For example, if the object first appears in a segment 1 in a first recording and then in a segment 4 of a second subsequent recording, i.e. the same tonal value in the histogram of a color channel appears significantly in each segment, then the object moves across the FOV of the camera unit (object tracking).

The method may further comprise issuing a prioritization command for increasing at least one parameter in connection with the recordings. The parameter may be increased such that a resolution of a recording section of the recordings corresponding to the sub-segment having the histogram with the temporally changed gradient is increased. Alternatively or additionally, further parameters may be adjusted, such as a sampling rate of the image processing unit and a recording speed of the camera unit.

A number of the plurality of sub-segments can be predetermined based on a computing power of the control unit.

The predetermined color channels can be all color channels of the sub-segment or a predetermined subset of color channels of the sub-segment. For example, with 255 classes or tone values, only every 15th class or every 15th tone value can be used.

The predetermined subset may be determined based on a predetermined application of the method.

How many sub-segments are created can depend on the computing power of the control unit, which can be an ECU or include one. The more sub-segments there are, the better the tracking (position of the segment in the overall picture) can be. The fewer sub-segments, the lower the computing effort can be and the smaller the amount of data that needs to be transmitted and stored.

Providing the histogram information may further comprise summarizing predetermined Histograms of the determined histograms for predetermined color channels of each sub-segment of the plurality of sub-segments. Different color channels may be combined into a common feature to further reduce required computational power. The aggregation may be determined based on an application of the method. The histogram information may characterize the aggregated histograms.

Providing the histogram information may further comprise reducing the determined histograms by means of a one-hot encoding method and a pulse code modulation, PCM demodulation, wherein the provided histogram information characterizes the reduced histograms.

Providing the histogram information can further comprise determining histograms of the determined histograms that are relevant to the application using a second trained machine learning algorithm that is designed to determine histograms that are relevant to an application. The second trained machine learning algorithm can be a neural network. The neural network can have been trained in such a way that classes or tonal values that are relevant to a respective application are determined.

The method may further comprise outputting object information that is output when determining the presence of the object in the at least one sub-segment. The object information may be output to another unit and/or a user in order to inform the unit or the user of the presence of the object. The object information may further characterize a direction of movement of the object.

The method may further comprise at least partially controlling an external entity, such as a vehicle, based on the object information. For example, a forklift may drive a different path based on information of obstacles along a predetermined path.

The object is achieved according to a second aspect by a system for determining an object in a recording, comprising an image processing module and a control device with respective means for carrying out the method according to the first aspect.

The system may further comprise a camera unit for recording the images and providing the images to the image processing module. The image processing module may be comprised of the camera unit or in a downstream and/or separate unit.

The controller may be or include an electric control unit, ECU. The controller may further include a memory for storing the first and/or second machine learning algorithm.

Method features described in relation to the method according to the first aspect may be embodied as features of the system according to the second aspect.

The object is achieved according to a third aspect by a computer program product comprising instructions which cause the system according to the second aspect to carry out the method according to the first aspect.

• 1 ein schematisches Ausführungsbeispiel eines Computer-implementierten Verfahrens zum Bestimmen eines Objekts in einer Aufnahme;
• 2 eine schematische Darstellung einer Segmentierung einer Aufnahme; und
• 3 ein schematisches Ausführungsbeispiel eines Systems zum Bestimmen eines Objekts.

Preferred embodiments are explained by way of example using the accompanying figures. They show

• 1 a schematic embodiment of a computer-implemented method for determining an object in a recording;
• 2 a schematic representation of a segmentation of a recording; and
• 3 a schematic embodiment of a system for determining an object.

In the figures, identical or essentially functionally identical or similar elements are designated by the same reference numerals.

1 shows a computer-implemented method 100 for determining an object in a recording A, which is carried out by means of a system according to the 3 is executable. In the further course of the description, the individual steps of the method 100 are therefore described with reference to respective units of the system.

The method 100 comprises providing 110 a plurality of image files characterizing a plurality of images A of a predetermined area taken one after the other to an image processing module 300. The predetermined area can be an area to be monitored, for example an industrial plant.

The plurality of image files can be recorded by a camera unit 200, in particular a recording unit 210 thereof. Furthermore, the plurality of image files can be transmitted to the image processing module 300. The image processing module 300 can receive the plurality of image files by means of a communication unit 340.

The image files of the multitude of image files can be individual frames of a video.

The method 100 further comprises providing 120, by means of the image processing module 300, histogram information to a control unit 400. For this purpose, the communication unit 340 of the image processing module 300 can send the histogram information to a communication unit 440 of the control unit 400 and the communication unit 440 of the control unit can receive the histogram information.

The provision 120 of the histogram information of the method 100 comprises at least the following steps 121, 122 for each of the images A. In step 121, the image is divided into a predetermined number of sub-segments T1-T4, as shown in the 2 is shown.

Furthermore, the providing 120 includes determining 122 a histogram for predetermined color channels of each sub-segment of the plurality of sub-segments. A histogram characterizes tone values and a number of tone values of the color channel of the sub-segment. Furthermore, the histogram information characterizes the determined histograms.

The tonal values of a histogram may be plotted along an X-axis and the number of tonal values may be plotted along a Y-axis of the histogram. The tonal values may be converted into classes for further processing to improve processing of the data. A histogram per color channel may have the values 0-255 on the X-axis in an RGB color space. The process of splitting the image and determining the histograms can be very computationally intensive, so these steps are performed in the image processing module 300.

To reduce the amount of data in the histogram information, for example, not every color channel can be used, but only every 15th color channel. This results in a significantly smaller amount of data. A smaller amount of data can lead to a lower resolution, so that depending on the application of the method, individual color channels can be viewed.

The steps of dividing 121 and determining 122 can be carried out by means of a controller 330 of the image processing module 300.

The method 100 further comprises generating 130, by means of the control device 400, a time-discrete signal based on the histogram information of the images taken one after the other. The generating step 130 can be carried out by means of the controller 430. Each of the controllers 230, 330, 430 can comprise a processor or be such a processor in order to execute corresponding computer program products.

The method 100 further comprises determining 140, by means of a first trained machine learning algorithm of the control unit 400, a presence of the object in at least one of the images A based on the time-discrete signal. The first machine learning algorithm is designed to determine the presence of the object based on a temporal change in a gradient of a tone value of at least one histogram of the histogram information. The first machine learning algorithm can be stored in a memory 420 of the control unit 400. Furthermore, the camera unit 200 and the image processing module 300 can have respective memories 220, 320 for storing information.

According to method 100, the gradients of the color channels for each sub-segment (gradient change) are observed over time. If a significant change in the frequency density for the intensity classes or tone values under consideration occurs in the histogram, then an object can be identified based on this. The proposed system architecture allows the presence of the object to be detected using a simple control unit, in particular an ECU. The control unit can be installed in a vehicle to be controlled, for example.

The histogram information can further comprise position information that characterizes respective positions of the plurality of sub-segments T1-T4 in the plurality of image files. The method 100 can further comprise, based on the position information, determining a position of the object in the at least one sub-segment T1-T4 having the histogram with the temporally changed gradient. Since it is known in which sub-segment T1-T4 this object appears, conclusions can be drawn about the position of the object in the overall image. If the object moves across several sub-segments T1-T4, the movement of the object in the overall image or the recordings A can be determined (“object tracking”).

Since it is known in which sub-segment T1-T4 the object was detected, a prioritization command can be used to increase at least one para meters in connection with the recordings are determined and output, for example to the camera unit 200 and/or the image processing module 300. The parameter can be increased such that a resolution of a recording section of the recordings corresponding to the sub-segment T1-T4 having the histogram with the time-varying gradient is increased. Consequently, an increased resolution can be required in the sub-segment in which the object was detected.

The memories 220, 320, 420 may be configured to store one or more computer program products. The controllers 230, 330, 430 may each be configured to execute the computer program products.

As already mentioned at the beginning, the 3 a possible system architecture, wherein the image processing module 300 is shown as a separate unit. Alternatively, the image processing module 300 can be included in the camera unit 200.

Reference symbol

100

Computer-implemented method for determining an object

110

Providing a variety of image files

120

Providing histogram information to a control unit using the image processing module

121

Dividing the recording into a predetermined number of sub-segments

122

Determining a histogram for predetermined color channels

130

Generating, by means of the control unit, a time-discrete signal

140

Determining, by means of a first trained machine learning algorithm of the control unit, a presence of the object

A

Recording

T1-T4

Subsegments

200

Camera unit

210

Recording unit

220

Storage

230

steering

240

Communication unit

300

Image processing module

320

Storage

330

steering

340

Communication unit

400

Control unit

420

Storage

430

steering

440

Communication unit

Claims (14)

Computer-implemented method (100) for determining an object in a recording, comprising the steps: Providing (110) a plurality of image files characterizing a plurality of recordings (A) of a predetermined area taken one after the other to an image processing module (300); Providing (120), by means of the image processing module (300), histogram information to a control device (400), wherein the provision (120) of the histogram information for two or more recordings of the recordings (A) comprises: Dividing (121) the recording (A) into a predetermined plurality of sub-segments (T1-T4); Determining (122) a histogram for predetermined color channels of each sub-segment of the plurality of sub-segments (T1-T4), wherein a histogram characterizes tone values and a number of tone values of the color channel of the sub-segment (T1-T4), wherein the histogram information characterizes the determined histograms; Generating (130), by means of the control device (400), a time-discrete signal based on the histogram information of the images (A) taken one after the other in time; Determining (140), by means of a first trained machine learning algorithm of the control device (400), a presence of the object in at least one of the images (A) based on the time-discrete signal, wherein the first machine learning algorithm is designed to determine the presence of the object based on a temporal change in a gradient of a tone value of at least one histogram of the histogram information. Procedure (100) according to Claim 1 , wherein the histogram information further comprises position information characterizing respective positions of the plurality of sub-segments (T1-T4) in the plurality of image files, wherein the method (100) further comprises: based on the position information, determining a position of the object in the at least one sub-segment (T1-T4) comprising the histogram with the time-varying gradient. Procedure (100) according to Claim 2 , further comprehensive: Determining a direction of movement of the object based on a temporal course of the position of the object in the subsegment (T1-T4). Procedure (100) according to Claim 2 or 3 , further comprising: issuing a prioritization command for increasing at least one parameter in connection with the recordings (A), wherein the parameter is increased such that a resolution of a recording section of the recordings (A) corresponding to the subsegment having the histogram with the time-varying gradient is increased. Method (100) according to one of the preceding claims, wherein a number of the plurality of sub-segments (T1-T4) is predetermined based on a computing power of the control unit (400). Method (100) according to one of the preceding claims, wherein the predetermined color channels are all color channels of the sub-segment (T1-T4) or a predetermined subset of color channels of the sub-segment (T1-T4). Procedure (100) according to Claim 6 , wherein the predetermined subset is determined based on a predetermined application of the method (100). The method (100) of any preceding claim, wherein providing (120) the histogram information further comprises aggregating predetermined histograms of the determined histograms for predetermined color channels of each sub-segment of the plurality of sub-segments (T1-T4). Method (100) according to one of the preceding claims, wherein the provision (120) of the histogram information further comprises reducing the determined histograms by means of a one-hot encoding method and PCM demodulation, wherein the provided histogram information characterizes the reduced histograms. Method (100) according to one of the preceding claims, wherein providing (120) the histogram information further comprises determining application-relevant histograms of the determined histograms by means of a second trained machine learning algorithm configured to determine application-relevant histograms. System for determining an object in a recording (A), comprising an image processing module (300) and a control device (400) with respective means (320-340; 420-440) for carrying out the method (100) according to one of the preceding claims. System according to Claim 11 , further comprising a camera unit (200) for recording the images and providing the images to the image processing module (300). System according to Claim 11 or 12 , wherein the control unit (400) is or comprises an electric control unit, ECU. Computer program product comprising instructions that cause the system to execute one of the Claims 11 until 13 the method (100) according to one of the Claims 1 until 10 executes.

Images