WO2023066435A1

WO2023066435A1 - Method and processor circuit for estimating an absolute area value of a free loading area and/or an absolute length value of free loading metres in a cargo space, and logistics system

Info

Publication number: WO2023066435A1
Application number: PCT/DE2022/200239
Authority: WO
Inventors: Carsten Segesser; Matthias Wagner
Original assignee: Continental Automotive Technologies GmbH
Priority date: 2021-10-20
Filing date: 2022-10-17
Publication date: 2023-04-27
Also published as: DE102021211827A1

Abstract

The invention relates to a method (35) for estimating an absolute area value and/or an absolute length value of free loading metres of a free loading area (12) in a cargo space (13), the free loading area (12) being an uncovered part of an overall rectangular total loading area (25) of the cargo space (13). According to the invention, image data (17) from a camera (16) are received by a processor circuit (15) and show a photograph (18) of the free loading area (12), wherein the free loading area (12) depicted in the photograph (18) is determined in the image data (17) by means of a segmentation algorithm (40), and, from the pixels (82) which show the segmented free loading area (12), associated absolute pixel area values (88) of the pixels (82) are added to form the absolute area value.

Description

Method and processor circuit for estimating an absolute area value of a free loading area and/or an absolute length value of free loading meters in a cargo hold and logistics system

The invention relates to a method and a processor circuit for estimating an absolute area value of a free loading area in a hold or cargo hold. An uncovered part of a total loading area of the freight compartment (freight compartment floor) is referred to here as "free loading area" and the real or absolute area value is an indication in square meters or square feet or similar, i.e. an indication in an absolute area measurement. Another advantageous dimension are the so-called free loading meters, which indicate the length of free loading space that can be derived from the area, for example, and that is guaranteed to be still available in the loading space. The estimate should be based on a photograph of the free loading area and loading meters. The invention also includes a logistic system using said processor circuit.

The loading status of truck trailers (trucks - trucks) or other cargo space of a logistics company is often unknown to fleet managers. As a result, fleet managers are unable to accept or allocate additional loads on the routes traveled by their trucks.

However, it is common for truck drivers to take photos of the interior of the cargo hold after loading goods or freight to document compliance with load securing. But these pictures do not give any information about the charge status.

A method for estimating a still remaining transport capacity in the cargo hold of a logistics system is known, for example, from EP 3 872 461 A1. The method uses image data from a photograph taken, for example, by a driver of the respective cargo hold. by means of one image processing algorithm, the information about the still available free loading area is determined from the image data.

EP 3 872 725 A1 also discloses a method for determining the loading status of a cargo compartment of a truck. Accordingly, a photograph of the cargo hold is used to determine where the freight items are located, and the distance from the front edge of the cargo hold at the entrance to the freight items is estimated using a grid projection. This requires the installation of a projector to project the grid projection in the hold.

The invention is based on the object of estimating the remaining free loading area in a freight compartment, for example in a trailer or box of a truck or in a freight container, with little outlay on hardware.

The object is solved by the subject matter of the independent patent claims. Advantageous developments of the invention are described by the dependent patent claims, the following description and the figures.

As a solution, the invention comprises a method for estimating an absolute area value of a free loading area and/or an absolute length value of the so-called free loading meters in a cargo hold. The uncovered cargo area is an uncovered portion of an overall cargo area cargo area. As already described, the term "absolute" refers here to a specification in an absolute measure of length or area. An absolute area value is therefore a specification, for example, in square meters or square decimeters or square centimeters or square feet, to name just a few. On the other hand, a relative area value would be an indication of a ratio, for example 50 percent or 70 percent. An absolute measure of length is accordingly a statement in meters or centimeters or feet, to name just a few examples. The cargo space whose available free loading area is to be estimated can be present, for example, in a trailer of a truck or the loading area of a truck or in a freight container. The free Loading area is that part on which, for example, pallets or freight items in general can be placed or parked on the floor of the cargo hold. If only the absolute length value is required (not also the absolute area value), then from such pixels, which form a vanishing line in the photograph, each of which has an upper edge (rear end or rear edge) and a lower edge (front end or front edge) of the connects segmented free loading area, associated absolute pixel depth values are added to the absolute length value of the free loading meters. However, if the absolute width value of the total loading area is known, the absolute length value can be calculated from the absolute area value instead.

The procedure uses the photograph described above, such as that which may have been taken by a truck driver, for example, to document the securing of the load. Based on these photographs or images, the free floor space (i.e. the free loading area) in the trailer or in the cargo hold in general is estimated with the help of image processing and/or artificial intelligence.

For this purpose, a processor circuit receives image data from a camera, which represents a photograph of the free loading area. The free loading area shown in the photograph is determined in the image data using a segmentation algorithm. The information is thus available as to which and/or how many pixels correspond to the free loading area. For those pixels that represent the segmented free loading area, associated absolute pixel area values of the pixels are summed to form the absolute area value. This gives you the information you are looking for about the free loading area as an absolute area information.

For example, a smartphone or a tablet PC or a smart watch or smart glasses (data glasses) can be used as a camera. There is therefore no need for a permanently installed sensor (eg camera, lidar, radar, ultrasonic sensor) in or on the cargo hold. Said processor circuit can be integrated in the camera, for example in a smartphone, or it can be a so-called backend server or a server computer which receives image data from the camera via a communication link, such as an Internet connection. A so-called segmentation can be carried out in such a photograph, i.e. a distinction or assignment as to what is represented by a single pixel of the photograph, i.e. whether the pixel is part of the image of the free loading area or part of an image of a freight item, to give just examples to call. After the application of a segmentation algorithm, there is information per pixel as to whether this pixel represents the free loading area or not. A very simple implementation of a segmentation algorithm can be based, for example, on an evaluation of color information in the photograph. If it is known that the entire loading area, i.e. the floor of the cargo hold on which freight items can be arranged, has a characteristic color, for example light gray or red or wood-colored, and the adjacent surroundings have a different color, then a pixel can be determined in the photograph whether it has that characteristic color, and then interpreting or associating that pixel as representing part or all of the vacant truck bed, which is referred to as segmenting. However, this is only an example; a preferred segmentation algorithm is described below.

The overall loading area can be assumed to be aligned or arranged horizontally. From a photograph of a horizontally arranged total loading area, geometric calculations can be used to determine how much absolute area the "image detail" corresponds to or represents of a single pixel. If you think of the pixel as a rectangular section of the photograph, the geometry of the camera and/or the optical system of the camera can be used to calculate a back projection, for example, which can be used to determine on which area in the hold or cargo hold this rectangle of a single pixel is projected would be displayed on the free loading area if you carry out the rear projection. The area size resulting from mapping, for example, said rectangle of the pixel onto the free loading area results in said absolute pixel area value, which is in a range from 5 cm ² to 0.5 m ² , for example can. If one summarizes all those pixels that are categorized or labeled or marked as free loading area according to the segmentation algorithm, then the sought-after one can be calculated by calculating or summing up their individual pixel area values.

Depending on the prerequisites or conditions under which the photograph was taken by the camera, it may already be known in advance how much area of the cargo hold is represented by a single pixel. A pixel represents a rectangular image section (the so-called "rectangle" in which a part of the scenery visible from the camera, in particular the free loading area, is depicted or displayed. Is it known how large (in terms of an absolute pixel area value) the area of the is the part of the free loading area represented by a single pixel, the total available free loading area can be calculated as an absolute area value by summing up all those pixels that belong to the free loading area according to the segmentation algorithm. For example, the absolute pixel area value per pixel can be specified in configuration data . An operator of the camera can then ensure that the camera is used to take the photograph from a perspective or from a position that results in a corresponding image of the free loading area, which is then converted or mapped into absolute pixel area values using the configuration data can become.

The invention has the advantage that no additional circuitry is required to be installed in the cargo hold itself in order to indicate the free loading area that is still available, i.e. that part of the total loading area that is not yet covered by a piece of freight, as an absolute area value, i.e. for example indicate that in the hold still 3 square meters or 7.5 square meters of free loading space are available. In addition, there is an advantage because the free loading area is actively searched for in the photograph/image data and not the other way around, freight items or blocking objects on the entire loading area. This simplifies the image processing in a decisive way, because the various possible shapes of freight items do not have to be taken into account.

The invention also includes developments that result in additional advantages.

A further development includes that it is assumed or given that the overall loading area is rectangular overall and in the photograph a front edge of the rectangular overall loading area is shown arranged parallel to a sensor plane of an image sensor of the camera and opposite side edges of the overall loading area in the cargo hold perpendicular to the front edge extend backwards. In other words, the floor of the cargo compartment is rectangular, as is usual for cargo containers or truck trailers. In order to ensure that the photograph represents or depicts the cargo hold with the entire loading area in the manner described, an instruction can be given or issued to the operator of the camera, for example. This can be achieved, for example, by means of an application program (a so-called app) that can be operated in the camera, for example a smartphone, in order to ensure that the operator takes the photograph from a corresponding perspective. The front edge is therefore arranged parallel to the sensor plane of the image sensor and is preferably aligned horizontally. The photograph is preferably taken centrally with respect to the front edge, so that, starting from the camera, a left-hand section and a right-hand section of the front edge are of the same length. The photograph is preferably taken from a height of between 1.4 meters and 2 meters above the entire loading area. The overall truck bed is preferably oriented horizontally while the photograph is being taken. The free loading area is preferably recorded from above at an angle from an area outside the overall loading area. The sensor plane of the image sensor is preferably arranged parallel to a vertical rear wall of the cargo hold that delimits the entire loading area at the rear. All of this helps avoid perspective distortion of the mapped edges and/or lines falsifying the estimate of the absolute area value. Corresponding instructions can be issued in the manner described via the camera, for example a smartphone, so that the operator behaves accordingly. In addition or as an alternative to this, at least one orientation line or guide line, preferably several guide lines, can be shown on a display (screen) of the camera, which must be aligned with the front edge and/or the side edges, for example, before the shutter button of the camera is pressed, i.e before the image data is generated. In addition or as an alternative, it can be provided that the camera triggers automatically if a match between the front edge and/or the side edges in relation to predetermined guide lines is detected by means of an image processing algorithm. All that is then necessary for this is that the operator aligns or pans or moves the camera and if the image processing algorithm recognizes that the front edge and/or the side edges match the orientation lines or guide lines, the photograph is then automatically generated or the camera is triggered.

Based on this, the method comprises that an absolute width value of a width of the entire loading area and thus a length of the front edge is received for the entire loading area. Such an absolute width value can be determined, for example, using a model specification of a model of the cargo hold, for example a model of a truck trailer. The absolute width value can also be read out, for example, from a standard that applies to this type of cargo hold, as is known, for example, in the case of cargo containers for shipping. The width value and/or model information relating to the cargo hold can be received, for example, via the camera, for example a smartphone, from the operator who took the photograph. In addition, the method includes that in the image data a respective progression line of the side edges is determined on the basis of the image data using a vanishing line detection algorithm. As already stated, in addition to the front edge, which is or runs horizontally and transversely in the picture or in the photograph, the photograph also contains part of the side edges that delimit the total loading area to the sides to the right and left and that go into the background of the picture are aligned towards. The side edges are in reality, that is, arranged parallel to the real total loading area. In photography, however, they are aligned to a vanishing point, as is known for the well-known perspective distortion or perspective alignment of photographs of a rectangle. Since the photograph is oriented obliquely from above onto the entire loading area, in particular the free loading area, the result is an arrangement of the side edges which, starting from the two ends of the front edge, taper upwards in a triangular manner. In a simple embodiment, the vanishing line detection algorithm can be based, for example, on the fact that the side edges are marked in color, for example they are distinguished from the surroundings by red adhesive strips or a previously known color or are shown in high contrast. The course of such a colored line in the photograph can then be detected in image processing by means of color comparison. However, this is just a simple example of a line of flight detection algorithm. An edge detection algorithm known per se can also be used.

In addition, it is intended that "pixel lines" will be determined in the photograph in the area of the segmented free loading area. These each consist of those pixels that extend from one of the gradient lines to the other of the gradient lines of the side edges to the other. An absolute pixel width value is determined for each pixel line from the absolute width value of the total loading area and the respective number of pixels in the pixel line. The triangle enclosed by the front edge and the progression lines of the side edges can be rastered line by line at least for a partial area in the photograph or in the image, so that horizontally running pixel lines result or these are defined or determined. Since the absolute latitude value is known, i.e. a value that can be in a range from one meter to eight meters, for example, and this absolute width value applies to each pixel line that connects both gradient lines (distance between the side edges), then by dividing the absolute width value by the Number of pixels of the respective pixel line can be determined, which absolute pixel width value is represented by the pixel, i.e. its horizontal absolute dimension represented on the free loading area. For example, a value in a range from 2 cm to 50 cm can result as a possible absolute pixel width value. This also depends on a pixel density of the described image sensor.

For the absolute pixel area value of the rectangle represented by the pixel on the free loading area, the second dimension is now missing, namely an absolute pixel longitudinal value or pixel depth value that indicates how far or how deep the pixel area represented by the pixel on the free loading area extends into the background or extending away from the camera. For this purpose, an imaging ratio is determined that indicates the size ratio in which a (in particular horizontal) line arranged parallel to the sensor plane is then imaged on the sensor plane in the real environment depending on its absolute distance from the camera and/or depending on a focal length of the camera . The line represents the width of a (real) object that is imaged on the image sensor. In a simple embodiment, the imaging ratio can be determined, for example, by experiments or test measurements, so that a characteristic diagram and/or a mapping table (lookup table) can be provided in the processor circuit. The imaging ratio can be used to decide how far an object or object or line must have been from the camera so that its width (i.e. the absolute width value) maps to or corresponds to a specific or given number of pixels.

The imaging ratio can be used, for example, to decide how far a line with a width of 2.5 meters must have been from the camera in order for it to have a width or length of 520 pixels in the photograph. The imaging ratio can thus be an indication X meters width is at A meters Spacing mapped to N pixels wide in the photograph. Since M and N are known, A can be calculated.

An absolute distance value of the pixel line can now be calculated for each pixel line using the absolute width value and the imaging ratio and the number of pixels in the pixel line. An absolute pixel depth value can be calculated from an absolute difference value of the distance value of the pixel line n and the distance value of an adjacent pixel line n+1. One can therefore assume that the absolute distance value of a row of pixels corresponds, for example, to the distance from the front edge of the rectangle represented by the pixel on the truck bed or from the rear edge of this rectangle. The absolute difference value of the absolute distance values of two adjacent pixel rows or pixel rows lying one on top of the other in the photograph can then be taken as an indication of the pixel depth or the length of the rectangle represented by the pixel in the direction of the image depth, i.e. perpendicular to the front edge of the entire loading area . This estimation has proven to be suitable to calculate a precise estimate of the free loading area as an absolute area value.

The product of the pixel width value and the pixel depth value corresponds to the absolute pixel area value of the rectangle represented by the pixel on the free loading area. Overall, an absolute pixel area value corresponding to the respective pixel can thus be calculated from the pixel width value and the pixel depth value for each pixel of the pixel rows and assigned to the pixel. Knowing the imaging ratio, it can thus be estimated without additional markings on the free loading area which absolute pixel area value is represented or imaged or shown by each pixel on the free loading area.

As an alternative, it can be provided to attach markers to the entire loading area, for example symbols of different colors, for example rectangles on the entire loading area. Such a marker can be detected by means of an image processing algorithm, for example based on its color value that gives rise to corresponding pixel values, and it can be counted by how many pixels are contained in the image of such a marker. Since the area of such a marker is known (e.g. the size of such a marker can range from 5 cm ² to 40 cm ² ), it can also be estimated what absolute pixel area value each of the pixels in the marker represents or maps. This pixel area value then also applies, at least approximately, to surrounding pixels. Multiple markers can be provided for different image depths.

A further development includes that the vanishing line detection algorithm includes that an edge detection is carried out in the photograph and detected edges are extended so that intersection points of the extended edges result, and a region with the highest density of intersection points is defined as the vanishing point of the photograph and the course lines are each used as a connecting line of the vanishing point to each end of the leading edge. This vanishing line detection algorithm has the advantage that it does not have to be known which of the edges shown in the photograph represent the actual side edges of the entire loading area. It is therefore not necessary to prepare the side edges, for example by color marking. It has been found that with the vanishing line detection algorithm described, the vanishing point in the image turns out to be the strongest or most probable intersection or most frequent intersection of the paired intersections of the extended edges, resulting in the highest intersection density value. The region may represent a single pixel or a group of multiple pixels, such as a 30 pixel by 30 pixel rectangle, to give just one example. It can then be evaluated for different regions how many intersection points are there. The region with the highest density of intersections (eg its center) then represents the vanishing point. As already described, the side edges in the geometric arrangement described point to this vanishing point. A line connecting this vanishing point with one end of the front edge of the total loading area then corresponds to the course of the two side edges. However, this connecting line is longer than the side edges, since the vanishing point may be in the area of the rear wall of the cargo hold and/or the side edges may be partially covered by cargo.

A further development includes that the segmentation algorithm includes the image data being supplied to a machine learning model as input data, with the model being trained to use a pixel-by-pixel classification of the image data to assign respective pixels to one of the following classes: the total loading area, an inner side wall of the Cargo hold, a ceiling of the cargo hold, and the model generates corresponding segmentation data, with a respective pixel of an unknown object leading to a segmentation result different from the classes. For example, it can be provided that training photographs are provided, to which so-called labeling data are additionally provided, which indicate where the total loading area, the inner side wall, the ceiling of the cargo hold are located in the respective training photograph. A training photograph can show an empty cargo hold or a loaded cargo hold. In a training photograph with a loaded cargo hold, the free loading area is then labeled or marked instead of the total loading area, i.e. the visible part of the total loading area. It has been found that a machine learning model can use such photographs to generate segmentation information or segmentation data that indicates where in a photograph of a loaded or unloaded cargo compartment the free loading area, inner side wall or ceiling is located. A different output value can be provided for the loading area, inner side wall and ceiling. For such a model of machine learning, it has been found that all other elements or objects, i.e. unknown objects (not loading area, inner side wall, ceiling) are identified by the model with a corresponding value or a corresponding segmentation result. For example, if the loading area is represented by a black pixel, the inner side wall is dark gray and the ceiling is light gray, the segmentation result for an unknown object can be white. An example of a suitable machine learning model is an artificial neural network. Appropriate training algorithms for training one Models in the manner described are known from the prior art. An alternative model can be what is known as a decision tree or a support factor machine (SVM), for example. The use of the segmentation algorithm with the machine learning model described to detect the free loading area, i.e. the free part of a total loading area of a cargo hold, represents a separate invention.

A further development includes that a triangular area from the progression lines of the side edges and the front edge of the cargo compartment floor (total loading area) is overlaid or masked with the segmentation data and the free loading area is defined as the intersection of the triangular area and the pixels of the "total loading area" class. Thus, the output of the segmentation algorithm, which delimits the pixels that belong to the loading area, i.e. the total loading area of the cargo compartment, laterally to the side by the progression lines and the front edge, which results in a more precise segmentation result or a more precise segmentation of the free loading area, since the Lines of progression and the front edge delimit the total loading area, while the segmentation data segments or marks the area of the total loading area recognizable or visible in the respective photograph, ie the free loading area. This can be used to address the problem that in the segmentation data, the edges of segmented regions, such as the total truck area, can be blurred.

A further development includes that the imaging ratio is the ratio

number of pixels in the pixel line / distance from the sensor plane to a lens of the camera absolute width value / distance from the line (ie the object) to the lens. The imaging ratio can thus be determined by calculation without prior tests or trials and thus without a table or characteristic map. The geometric calculation described here is based on the optical conditions and the law of rays. The distance of the sensor plane to the lens can be determined, for example, from the model of the camera or the camera type, for which purpose the metadata available in the image data can advantageously be used, which usually also indicate the type of lens in addition to the camera type.

A development includes that the imaging ratio is determined based on a focal length value of the photograph and/or based on a characteristic map of the camera. The map described here is the storage of those data that can be generated by experiments to empirically determine the imaging ratio. The focal length value can also be used to determine which imaging ratio results, namely, for example, the magnification factor.

The following image ratio can also be derived from further geometric calculations based on the course of the rays of the light:

Width of the object (i.e. length of the line) on the sensor / focal length f = absolute width value of the object (i.e. length of the line) / distance to the lens.

A sensor dimension for converting a width of an image of an object on the sensor into a corresponding number of pixels can be determined, for example, using the camera type/camera model of the camera used or from an aspect ratio and a crop factor of the image sensor of the camera. The width of a pixel is also referred to as the pixel pitch, i.e. an absolute distance and thus an effective absolute width of a pixel in millimeters or micrometers.

Another calculation option is given by

1 / focal length f

1 /distance sensor plane to lens + 1 /distance of object to lens. As a specific implementation, for example, based on the above equations, it can be provided that for a pixel line that connects the said progression lines with their known absolute distance value (= absolute width value of the total loading area), the number of pixels is counted and the absolute imaging width based on the calculation of the object on the image sensor

Total sensor width in mm * number of pixels in the pixel line / sensor total width in pixels the absolute width value of the pixel line in millimeters (mm) is determined as an intermediate result and thus the distance between the displayed line, which is represented by the pixel line: absolute distance value of the object (horizontal line between the side edges) to the lens absolute width value (e.g. 2.5 meters) * focal length f / absolute imaging width of the object on the image sensor.

A development includes that the imaging ratio is determined from predetermined meta information contained in the image data, in particular from EXIF data (exchangeable image file format). As already explained, the model of the camera and/or a lens used can be determined from the meta information. In addition, the focal length f used is also available in meta information, in particular the EXIF data. Additionally or alternatively, depth value data can be used, such as are known, for example, in connection with the so-called portrait mode of cameras. A TOF sensor (TOF—Time of Light) can be used here, the depth values or depth data of which indicate the distance between an image point or pixel and the camera. A development includes that the absolute width value is determined from type data of a container type of the cargo hold. The type data of such a container type can be determined, for example, from design data of the cargo hold.

A further development includes detecting edges of a rectangular entrance opening of the cargo hold as a rotated rectangle in the image data and straightening the image data by means of an image rotation and/or cropping them to the rectangular entrance opening by means of a section calculation. In other words, the entire entrance area or frame of the cargo hold, for example the door frame or gate frame of a truck trailer or the entrance of a freight container, can be depicted in the photograph. This results in the advantage that a rectangular shape is present in the photograph, which can be detected using an algorithm that is known per se. This can be used to ensure that the leading edge is mapped as a horizontal line. In addition, the frame of the entrance to the cargo compartment can be used to identify which part of the photograph or image represents the interior of the cargo compartment or the interior of the cargo compartment.

A further development comprises that the freight compartment of a truck or a freight container or a suitcase of a truck or a transport vehicle or a goods wagon or a ship or a transport aircraft is depicted in the photograph. The method has proven to be particularly robust and reliable for these two types of cargo holds.

As a further solution, the invention comprises a processor circuit set up to carry out an embodiment of the method. The processor circuit can, for example, be operated in a stationary manner in a logistics company or in a data center of a logistics company. The processor circuit may include one or more microprocessors, which may be coupled to a data memory in which Program instructions can be stored which, when executed by the processor circuit, cause it to carry out the embodiment of the method. The image data of the photograph can be received by the processor circuitry from the camera, for example via an Internet connection or a communication link. Of course, the processor circuit can also receive image data from a number of cameras, in order to be able to analyze the free loading area of a number of freight compartments with regard to the absolute area value in the manner described. The total free loading area available in several cargo holds can thus be determined by the processor circuit.

As a further solution, the invention comprises a logistics system with an embodiment of the processor circuit and with a receiving device for receiving image data from photographs of different cargo compartments, the logistics system being set up to use the processor circuit to determine a respective free loading area of the cargo compartment and a freight distribution dependent thereon of items of freight to be transported. Thus, the transport or the distribution of freight items to a plurality of holds can be carried out by the logistics system on the basis of the processor circuit and thus on the basis of an embodiment of the method according to the invention.

The invention also includes the combinations of features of the described embodiments.

An exemplary embodiment of the invention is described below. For this shows:

1 shows a schematic representation of an embodiment of the logistics system according to the invention with a processor circuit;

FIG. 2 shows a flow chart of an embodiment of the method according to the invention, as it can be executed by the processor circuit of the logistics system of FIG. 1; FIG. 3 sketches to illustrate method steps of the method from FIG. 2;

4 shows a sketch to illustrate a calculation of pixel area values of pixels of a photograph showing a free loading area of a cargo hold; and

5 shows a sketch to illustrate the method steps for calculating the free loading meters.

The exemplary embodiment explained below is a preferred embodiment of the invention. In the exemplary embodiment, the described components of the embodiment each represent individual features of the invention to be considered independently of one another, which also develop the invention independently of one another and are therefore also to be regarded as part of the invention individually or in a combination other than the one shown. Furthermore, the embodiment described can also be supplemented by further features of the invention already described.

Elements with the same function are each provided with the same reference symbols in the figures.

1 shows a logistics system 10, by means of which it can be monitored for a truck fleet with at least one truck 11, how much free loading area 12 is still available in the cargo compartments 13 of the trucks 11. Only one truck 11 is shown in FIG. 1; ellipses 14 are intended to indicate that the truck fleet may also include other trucks 11. The logistics system 10 can have a processor circuit 15 to determine the total available free loading area 12 per truck 11 . The processor circuit 15 can receive image data 17 from at least one camera 16, which shows a photograph 18 of the cargo hold 13. The image data 17 can, for example, via the Internet 19 and/or a Cellular network 20 are transmitted from the camera 16 to the processor circuit 15, for example by means of Internet connections 21 known per se. The camera 16 can be implemented by a smartphone 22, for example. For example, a driver of the respective truck 11 can use the camera 16 to take the photograph 18 of the cargo hold 13 . Each driver can use his own smartphone 22 for this, for example. The photograph 18 can be taken in such a way that a front edge 23 is shown as a horizontal line in the photograph 18 and the camera 16 is held above the free loading area 12 such that the free loading area 12 is visible from a perspective obliquely from above , as shown in FIG. In the example shown, the cargo hold 13 is already partially loaded with cargo items 24, which can be pallets, for example. The free loading area 12 can be part of a total loading area available in the loading area 13 or the total loading area 25, which can be formed by the floor of the loading area 13, for example. It is also shown how the total loading area 25 can be delimited by side edges 26. It is also shown how side inner walls 27, a ceiling 28 and a rear wall 29 can also be seen in the photograph 18. For example, the photograph 18 may be photographed from outside of the cargo hold 13 through a door frame 30 , which may also be depicted in the photograph 18 such that it is shown as an outer rectangle enclosing the cargo hold 13 .

The logistics system 10 can use the processor circuit 15 to calculate an absolute area value from the received image data 17, which can indicate how many square meters or square decimeters or square feet of free loading area 12 is still available in the respective cargo hold 13. Shown is how it can be determined for an individual truck 11, which can be represented in the processor circuit 15 by its vehicle ID V-ID, to what extent the loading capacity CAP in the freight compartment 13 has already been exhausted and how much free loading capacity is still available is available. Since an absolute area value is determined, the footprint for additional pallets 33 can be determined accordingly, which in FIG additional pallets 33 can result. 1 also shows how a method 35 can be used to determine where the free loading area 12 can be found on the total loading area 25 . By means of an additional algorithm, there can also be a half-loaded loading area 32 which is covered by freight items but still has space towards the ceiling 28 .

The method 35 for finding or determining the free loading area 12 in the cargo hold 13 is described below with reference to FIG. 2 and illustration by means of FIG.

It is assumed here that a freight compartment 13 was loaded with freight items 24 as a start SC and that in a step S11 an operator, for example a driver of the truck 11 , takes the photograph 18 using a camera 16 . In a step S12, the image data 17 of the photograph 18 can be transmitted to the processor circuit 15 of the logistics system 10 in the manner described. The image data 17 of the photograph 18 is preferably color image data, ie an RGB image (RGB—red, green, blue). It can be stored in the image data 17 in JPG format, for example (JPG—Joined Photographic Expert Group).

Image pre-processing can be carried out in a step S13 in order to emphasize edge profiles, for example, as can be effected with the “unsharp masking” algorithm, for example.

In a step S14, a segmentation algorithm 40 can be executed. For example, the image data 17 resulting from step S13 can be transferred as input data to a machine learning model 41, for example an artificial neural network. The model 41 may perform pixel-by-pixel segmentation or categorization of the photograph 18 pixels. 3 shows a segmentation result as segmentation data 42, which can contain segmentation results 25' for recognizing the entire loading area 25, 27' for recognizing inner side walls 27 and 28' for recognizing the ceiling. A segmentation result 43 a unknown object can result, for example, from the freight items 24 and/or the rear wall 29 . Shown is that boundaries or edges may be fuzzy or fuzzy as may result in a machine learning model. Step S13 can also follow step S14.

The segmentation algorithm can also provide a binary segmentation 50, in which the classes total loading area, inner side walls and ceiling from the segmentation data 42 are assigned a first segmentation value 51 and all other segmentation results 43 receive a second segmentation value 52, for example 1, i.e. different from the first segmentation value 51 . This results in a total segmentation mask 53.

For example, step S15 may include a rotation such that the door frame 30 is represented as an upright rectangle such that the leading edge 23 in the rotated photograph represents a level or horizontal line. In addition, a section calculation 54 can be carried out, which provides for image areas outside of the door frame 30 to be cut away, so that the only remaining image content is through the door frame 30, ie the view into the cargo hold 13 . The door frame 30 represents an entrance opening of the cargo hold 13.

In a step S15, the image section from the section calculation can be accessed

54 an edge detection can be used, by which edges recognizable in the photograph 18 in the cargo hold 13 are detected. Each detected edge can be extended to span the entire image.

This results in points of intersection 55, only some of which are provided with a reference symbol for the sake of clarity.

It has been found that the points of intersection 55 accumulate or concentrate in the area of the vanishing point 56 of the photograph 18 . In other words, there is a region 57 into which there is a density or frequency of intersections

55 is largest, which corresponds to the vanishing point. This can for example can be recognized by means of a histogram display 58, which indicates for coordinates X, Y in the image or photograph 18 for individual regions how many points of intersection 55 were respectively detected therein. As a result, the region 57 can be detected as the region with the greatest spatial or local density of intersection points 55 .

3 further illustrates how a triangle 60 can be determined that connects the determined vanishing point 56 and outer ends 61 of the front edge 23 of the overall loading area.

The segmentation algorithm 40 can further comprise a step S16 according to which the triangle 60 and the binary segmentation 50 are overlaid or combined as a mask. For this purpose, the triangle 60 can also be defined as a binary value, so that a pixel-wise overlay or combination is possible, but other geometric calculations for the combination of the area of the triangle 60 and the binary segmentation 50 can also be used. In this way, that area 62 is identified or segmented that lies within the triangle 60 and thus corresponds to the running edges 63 of the side edges 26 and is closed at the front by the front edge 23 . This corresponds to an estimate of the total truck area plus a portion of the rear wall 29 up to the vanishing point 56. When this is overlaid with the binary segmentation 50, the portion of the triangle 60 that the machine learning model 41 classifies as "total truck area" ( i.e. "cargo floor") duly combined. The superimposition result 64 can thus be used to detect the free loading area 12 . In this case, however, the fuzzy edges from the segmentation data 42 are eliminated.

FIG. 3 shows how the free loading area 12 can be identified or marked as a pixel set 65 in a step S19 in the photograph 18 or in the image detail of the loading space or cargo space 13 . Fig. 4 illustrates how, starting from the pixel set 65, i.e. from those pixels in the image data 17 which represent the free loading area 12, it can be determined which of the absolute or real area or which absolute area value the free loading area 12 corresponds to. The left-hand side shows how geometric calculations based on image data 17 of photograph 18 can be used to determine the width of a horizontal line 70 with a known absolute width or an absolute length value by imaging light rays 71 via a lens system 72 of the camera 16 on a sensor plane 73 of an image sensor 74 of the camera 16 leads to a line image 70' with a resulting image length (specified, for example, in a number of pixels). This results in an overall imaging ratio 75 which is dependent on a distance value 76 of the respective line 70 from the camera 16 . An example for the calculation of the distance value 76 based on an image length of an imaged line or line image 70' of a horizontal line 70 is known, for example, on the website www.lensation.de/calculator.html.

In Fig. 4 on the right side it is shown that this can be used in a step S17 to calculate horizontal lines, such as the front edge 23, with a known absolute width value 80 of the front edge 23 and thus the width of the total loading area 25 to a real or to infer or calculate absolute distance value 76 . 4 illustrates how the free loading area 12, ie the pixel set 65, can be divided into individual pixel rows 81 of the pixels 82 arranged horizontally next to one another. For the sake of clarity, only a few pixels 82 and individual pixel lines 81 are provided with a reference number. An absolute pixel width value 84 can be calculated for each pixel 82, which indicates how wide the portion of the free loading area 12 represented by the pixel 82 is. This can be calculated by dividing the absolute width value 80 of the width of the total truck bed by a number of pixels of the respective row of pixels 81, giving the absolute or real pixel width value 84. For each pixel line 81, the absolute distance value 76 for the pixel line 81 can be inferred or calculated on the basis of the number of pixels 82 contained therein and the imaging ratio 75 become. This absolute distance value 76 can be interpreted, for example, as the distance of a pixel leading edge 85 of the pixel. If one also takes the absolute distance value 76 of the following pixel line n+1 for each pixel line n, then the distance of the pixel trailing edge 86 can be estimated for the absolute distance value 76 of the pixel line n+1. Overall, this results in an absolute pixel depth value 87 as the difference between the absolute distance values of the pixel leading edge 85 and the pixel trailing edge 86.

The product of the absolute pixel width value 84 and the absolute pixel depth value 87 then results in a pixel area value 88 of a pixel area of a respective individual pixel 82. An absolute pixel area value 88 can therefore be assigned to each pixel 82 that represents the free loading area 12, i.e. to each pixel 82 of the pixel set 65, an absolute be assigned pixel area value 88.

By summing up all the pixels 82 of the free loading area 12 in a step S18 (Fig. 2), the absolute area value of the free loading area 12 results, as determined from the photograph 18, i.e. its image data 17, without additional technical aids in the cargo hold 13 or can be calculated.

The absolute width value 80 and/or the imaging ratio 75 can be determined, for example, by configuring the processor circuit 15 and/or by an input from an operator.

Fig. 5 illustrates how, using the known absolute pixel depth value 87 from a row of pixels 100, each lined up along a predetermined alignment line 101, by adding up the pixel depth values 87 of the respective pixel line, a respective absolute length value of the free loading path or free loading meter 102 can be calculated. The respective vanishing line 101 can be arranged at predetermined positions 103 of the segmented free loading area 12, eg middle, middle left half, middle right half. The free loading area (12) is preferably divided into several tracks (in particular either 3 tracks or 2) in order to determine how many pallets can be placed in the remaining space, ie a vote can be taken take place on a planned pallet width. The respective vanishing line 101 connects a front edge 104 of the segmented free loading area 12 arranged at the bottom in the photograph, i.e. usually the front edge 23 in the case of error-free segmentation, with a rear edge 105 of the segmented free loading area 12 arranged at the top in the photograph, as shown from the binary segmentation 50 can result. The respective vanishing line 101 points to the vanishing point 56 .

The main benefit is that there is no need to install switching hardware in the trailer or on the cargo hold to determine the free capacity / free loading space.

Estimating the remaining vacant cargo space in a trailer or other transport container using machine vision requires only a view of the cargo space that is still vacant. The best field of view would be a shot from inside the trailer. But each camera may have a different lens, so we may be missing an estimate of free space. You must be outside the trailer in a position where you cannot see the ground plane. It is almost impossible to estimate occupancy based on roof level, as there is no indication of the height of the pallet.

The solution to the fleet manager's problem may be a camera-based machine learning algorithm that detects empty cargo capacity. This algorithm can be run on the device of the truck driver or the shipper (the shipper is a person who loads the truck in a logistics center) (e.g. a smartphone) or in the cloud based on an image generated by any device capable of capturing an image is uploaded. This approach eliminates the need to add an additional sensor to the trailer or loading dock. It is completely new to consider the driver and their smartphone as part of the system, which means that no additional sensors are required to assess the load capacity. This information can be passed on to the fleet manager to assess whether or not additional loads can be accepted. After loading the trailer, the truck driver takes a photo to prove that the load has been secured. A picture of the rear of the trailer is entered (taken to demonstrate compliance with the load securing guidelines).

For trailers loaded from the side, the image could be taken from the side accordingly. Compared to the previously proposed solution, the algorithm is not limited to detecting free hold or cargo space between a detected object (e.g. a pallet) and the beginning of the trailer, but can directly segment the image and decide for each pixel whether it is classified as free ground space.

This results in the solution being able to detect free loading meters of trailers with any type of load. From this image, the free capacity of the trailer is calculated as described below.

The algorithm:

Input: o Image of the CargoBox (freight compartment) o Configuration:

focal length f of the camera

Camera crop factor

Cargo box width [m]

The main component of the algorithm is a segmentation model that can be configured as follows, for example: o RGB image as input o Transforms the visible floor (free loading area) of the cargo box into a first pixel value, e.g. (0,0,0) o Transforms the side walls of the cargo box into a second pixel value e.g. (63, 63, 63) o Transforms the cargo box top into a third pixel value e.g. (127, 127, 127) o Transforms the cargo box back wall into a fourth pixel value e.g. (191 , 191 , 191 ) o Converts all other fourth pixel values e.g. (255, 255, 255 )

The largest contour of the segmentation (corresponding to the entrance opening into the cargo compartment) is extracted and the minimum rotated rectangle around the contour is calculated.

-> Input image is rotated ( -angle of rotation of the rectangle)

-> Rectangle is cut out from the input image (width and height of the rotated rectangle)

-> Cargo Box becomes the still viewed image section

The vanishing point is detected on the image of the cargo box: o Line detector detects all lines with the corresponding angle and length in the image o Corresponding lines are enlarged to the image size of the cargo box o The intersection points of all lines are determined o Each intersection point in the image is identified using the Using the following formula in a triangle: 1/focal length f = 1/image distance + 1/object distance, the distance of each pixel within the triangle can be determined

By using charge width/number of horizontal pixels, the actual width of each pixel within the triangle can be determined. With the distance and depth, the area of each pixel within the triangle can be determined. The segmented image is binarized with a threshold (e.g. < 63).

This returns the bottom of the cargo box. For each remaining/segmented pixel its pixel area is added. This results in the remaining free space, i.e. the visible free loading area. The free space is preferably divided into several lanes (either 3 lanes or 2) to determine how many pallets can be placed in the remaining space. General advantages are: The missing information about the load condition of the trailer can be obtained from images taken to demonstrate compliance with load securing regulations. No additional process steps are required. This simplifies the work processes for truck drivers and saves time. There is no additional hardware to purchase, install, ship or maintain. Therefore, the proposed solution is significantly cheaper compared to the prior art. Reducing empty kilometers for trucks and improving capacity utilization result in fewer unnecessary trips and less CO2 per transported goods. In times of growing awareness of climate change, this is a major advantage that should not be underestimated.

Technical advantages are also: When using the segmentation model, the training effort is significantly lower, while the calculation results are significantly more robust. The model does not have to be trained for all types of cargo that could be in the trailer, but instead directly classifies pixel by pixel whether the pixel is classified as free trailer floor space.

Overall, the example shows how a camera-based determination of a loading status (available free loading space) in a cargo hold can be provided using segmentation.

Reference List

10 logistic system

11 trucks

12 loading area

13 cargo hold

14 ellipses

15 processor circuit

16 camera

17 image data

18 photography

19 Internet

20 cellular network

21 internet connections

22 smart phone

23 leading edge

24 pieces of cargo

25 total cargo area

26 side edge

27 side inner walls

28 ceiling

29 rear panel

30 door frames

32 loading area

33 pallets

34 payload value

35 procedures

40 segmentation algorithm

41 model

42 segmentation data

43 segmentation results

50 binary segmentation

51 segmentation value Segmentation mask Section calculation Intersections Vanishing point Region Histogram display Triangle Ends Area Progressive edges Overlay result Pixel quantity Line Light beams Lens system Sensor plane Image sensors Imaging ratio Distance value Width value Pixel line Pixel Pixel width value Pixel front edge Pixel rear edge Pixel depth value Pixel area value Row Alignment line meter Position Lower edge Upper edge

Claims

patent claims

1 . Method (35) for estimating an absolute area value of a free loading area (12) and/or an absolute length value of free loading meters in a cargo hold (13), the free loading area (12) being an uncovered part of a total loading area (25) of the cargo hold (13) is, d a r c h g e n n z e c h n e t , that image data (17) of a camera (16) are received by a processor circuit (15), which image data (17) represent a photograph (18) of the free loading area (12), wherein in the image data (17) by means of a segmentation algorithm (40) the free loading area (12) depicted in the photograph (18) is determined and absolute pixel area values (88) of the pixels (82) assigned to those pixels (82) which represent the segmented free loading area (12) are summed to form the absolute area value and/or absolute pixel depth values (87) assigned from those pixels (82) which form a respective vanishing line (101) connecting an upper edge (105) and a lower edge (104) of the segmented free loading area (12) in the photograph (18) added up to the absolute length value of the free loading meters (102).

2. The method (35) according to claim 1, wherein it is assumed that the overall loading area (25) is rectangular overall and in the photograph (18) a front edge (23) of the rectangular overall loading area (25) is parallel to a sensor plane (73) of an image sensor (74) of the camera (16) and opposite side edges (26) of the overall loading area (25) in the cargo hold (13) extend perpendicularly rearward from the front edge (23), and to the overall loading area (25) an absolute width value (80) a width of the entire loading area (25) and thus a length of the front edge (23) is received and in the image data (17) a respective progression line (70) of the side edges based on the image data (17) by means of a vanishing line detection algorithm and Pixel rows (81), which each consist of those pixels (82) that extend from one of the progression lines (70) to the other of the progression lines (70), are determined and per pixel row (81) from the absolute width value (80) and an absolute pixel width value is determined based on a number of pixels (82) in the pixel line (81) and a reproduction ratio (75) is determined, which indicates the size ratio of a line (70) arranged parallel to the sensor plane (73) as a function of its absolute distance (76) to the camera (16) and/or depending on a focal length (f) of the camera (16) on the sensor plane (73), and per pixel line (81) an absolute distance value (76) of the pixel line (81) is calculated based on the absolute width value (80) and the imaging ratio and the number of pixels (82) of the pixel line (81), and from an absolute difference value of the distance value (76) of the pixel line (81) and the distance value (76) of an adjacent pixel line (81) the absolute pixel depth value (87) is calculated and for each pixel (82) of the pixel rows (81) the absolute pixel area value (88) corresponding to the respective pixel (82) is calculated from the pixel width value (84) (80) and the pixel depth value (87) is calculated and assigned to the pixel (82).

The method (35) of claim 2, wherein the vanishing line detection algorithm comprises performing edge detection in the photograph and extending detected edges to yield intersections (55) of the extended edges, and a region (57) of highest intersection density as Vanishing point (56) of the photograph is set and the course lines (70) are each set as a connecting line (70) of the vanishing point (56) to one end of the front edge (23).

4. The method (35) according to any one of the preceding claims, wherein the segmentation algorithm (40) comprises that the image data (17) a model (41) of machine learning are supplied as input data, wherein the model (41) is trained to do so by means a pixel-by-pixel classification of the image data (17) to assign respective pixels (82) to one of the following (61) classes: the total loading area (25), a side inner wall and/or a ceiling (28) of the cargo hold (13); and the model (41) generates corresponding segmentation data which assign individual pixels (82) of the photograph to one of the classes, with a respective pixel (82) of an unknown object depicted in the photograph leading to a segmentation result which differs from the classes.

5. The method (35) according to claim 4 in its dependent reference to claim 3, wherein a triangular area from the course lines (70) and the leading edge (23) is masked with the segmentation data (42) and the free loading area (12) as the intersection of the triangular area and the pixels (82) of the class "Total loading area (25)".

6. The method (35) according to any one of claims 2 to 5, wherein the imaging ratio (75) is the ratio

Number of pixels (82) of the pixel line (81)/distance of the sensor plane (73) to a lens (72) of the camera (16) absolute width value (80)/distance (76) of the line (70) to the lens or

Width (70') of the object on the sensor / focal length (f) absolute width value (70) of the object / distance (76) to the lens, or

1 /focal length (f)

1 /distance sensor plane to the lens + 1 /distance (76) of the line to the lens.

7. The method (35) according to any one of claims 2 to 6, wherein the imaging ratio (75) is determined using a focal length value (f) of the photograph and/or using a characteristic map of the camera (16).

8. The method (35) according to any one of claims 2 to 7, wherein the imaging ratio (75) is determined from the image data (17) contained in predetermined meta information, in particular from EXIF data.

9. The method (35) according to any one of claims 2 to 8, wherein the absolute width value (80) from type data of a container type (41) of the cargo hold (13) is determined.

10. The method (35) according to any one of the preceding claims, wherein edges of a rectangular entrance opening of the cargo hold (13) are detected as a rotated rectangle in the image data (17) and the image data (17) are straightened by means of an image rotation and/or by means of a section calculation (54) trimmed to the rectangular entrance opening.

11. The method (35) according to any one of the preceding claims, wherein the cargo hold (13) of a truck (11) or a freight container or a transport vehicle or a goods wagon or a ship or a transport aircraft is depicted in the photograph.

12. Processor circuit (15) which is set up to carry out a method (35) according to one of the preceding (61) claims.

13. Logistic system (10) with a processor circuit (15) according to claim 12 and with a receiving device for receiving image data (17) of photographs of different freight compartments (13), wherein the logistic system (10) is set up to use the processor circuit (15) to determine a respective free loading area (12) of the freight compartment (13) and to control a freight distribution of freight items (24) to be transported (61) on this basis.