EP3696135B1 - Forklift and system with forklift for the identification of goods - Google Patents

Description

The invention relates to a forklift truck and a system with a forklift truck for identifying goods to be picked up / picked up. A forklift truck according to the preamble of claim 1 is from the EP0254192 A2 known.

The handling of empty beverage bottles has been regulated within Germany for some time. In order to save energy and resources, glass and plastic bottles are reused by means of a deposit system. In 2003, a deposit was therefore introduced for one-way bottles, after the corresponding numbers and the reusable rate had fallen over the years. This counteracted the high volume of waste.

In the reusable system, bottles are used that can be reused immediately for filling after cleaning and testing. In direct contrast to this, there is the one-way system, the products of which are broken down into their basic components after being taken back and processed further, for example into glass or plastic granulate.

According to studies by the Federal Environment Agency, returnable bottles are more environmentally friendly than single-use bottles. The energy and resource consumption for return transport and cleaning is lower for returnable bottles than the additional manufacturing costs for disposable bottles. This applies all the more, the more regional the distribution and the higher the number of refills.

A distinction between reusable bottles and one-way bottles in terms of shape and quality is usually only possible with plastic bottles (mostly made of the material PET, polyethylene terephthalate). In comparison to the one-way bottles, reusable bottles are characterized by a higher wall thickness and thus stability, since the bottles have to withstand a significantly longer life cycle.

• Externes Leergut-Management (bei den Kunden eines Getränkeherstellers)
• Internes Leergut-Management (beim Getränkehersteller):

In the case of returnable bottles, regardless of the material of the bottles, the so-called empties management poses major challenges for all beverage manufacturers. In principle, empties management can be divided into two different sub-areas:

• External empties management (for the customers of a beverage manufacturer)
• Internal empties management (at the beverage manufacturer):

The external empties management includes the actual acceptance of empties within the beverage trade via the customers, the local or external pre-sorting and the logistical link or the transfer of the empties to the destination of the beverage manufacturer.

In contrast, the internal empties management includes the delivery of empties to the beverage manufacturer itself, the data recording and storage of the goods, cleaning and inspection, as well as the appropriate provision of the empties at a suitable point in production.

In the following, the problem is explained using the empties management process.

The used reusable bottles returned by customers are usually pre-sorted at the dealer's location or by special service providers. This pre-sorting is largely done manually in the beverage return points or also automatically by the beverage return machines installed in the retail trade. With these service providers, this pre-sorting is also carried out manually or automatically. These processes of pre-sorting in the external empties management ensure a relatively high degree of purity of the beverage crates, so that it can be assumed that the correct bottles are contained within a crate.

After the empty returnable bottles have been transported to the beverage manufacturer, the internal empties management starts.

The delivery of empties to the beverage manufacturer is usually done in the form of boxes on standardized, reusable transport pallets. The amount of empties that has to be processed here is very high for a large beverage manufacturer. For example, around 30,000 trucks with 36 pallets each are processed at the Gerolsteiner mineral spring.

The process in the prior art is as follows:
The forklift drivers have to unload the truck with very short deadlines. While the forklift driver takes the pallets on the fork, he usually has to record the number of pallets and the crate / bottle type using a terminal in the forklift. This data then flows directly into the merchandise management system, which provides the data for the subsequent storage and production processes.

• Kurze Zeitvorgaben für die Entladung
• Mehrere Stapler entladen einen LKW, dadurch Konzentration auf Staplerverkehr
• Die Stapler können (bei entsprechender Ausstattung) mehrere Paletten gleichzeitig auf die Gabeln nehmen. Dies sind oft 4 Paletten, auch 8 oder bis zu 12 Stück.
• Aus Marketinggründen bestehen inzwischen sehr unterschiedliche Kasten-/Flaschentypen, welche leicht verwechselt werden können
• Schlechte Sichtverhältnisse bei der Entladung
• Manuelle Eingaben in aufgrund der hohen Sortenzahl unübersichtlichen Menüs
• Unterschiedliche Qualität der Vorsortierung, d. h. auf einer Palette sind unterschiedliche Kasten-/Flaschentypen enthalten

The detection of the crate / bottle type is highly error-prone. The reasons for this high error rate include the following:

• Short deadlines for unloading
• Several forklifts unload a truck, so concentration on forklift traffic
• The forklifts can (with the appropriate equipment) take several pallets onto the forks at the same time. This is often 4 pallets, also 8 or up to 12 pieces.
• For marketing reasons, there are now very different types of crates / bottles, which can easily be mixed up
• Bad visibility during unloading
• Manual entries in menus that are confusing due to the large number of types
• Different quality of pre-sorting, ie different crate / bottle types are contained on one pallet

Since the input quality of the forklift driver directly influences the expected actual stocks of empties, excessive deviations can lead to direct consequential problems in production. For example, the production of a batch of a product with the corresponding bottle type is started. Because of the wrong stocks, the wrong crate / bottle types are initially processed in the production-accompanying logistics and then the incorrect assignment is only recognized during the production process. In addition, the planned crate / bottle quantity may not physically exist at all (discrepancy in inventory) and production will be stopped prematurely. The system has to be converted and the planned product cannot be produced in the required quantity. This causes high costs in the beverage industry, which is already under pressure due to the price war in the retail sector. In addition, there is a considerable waste of resources and energy due to the avoidable expenses for handling the wrong crates / bottles.

The following example is intended to illustrate the relevance of the existing problem:

With up to 6,000 pallets per day and an error rate of up to 10% (ie 600 pallets) this would mean a misallocation of 480,000 bottles. Viewed over several days, this leads to a completely inadequate recording of the actual inventory of empties. It is assumed here that completely single-type pallets with only one type of crate / bottle have already been provided in the upstream external empties management. Particularly with a high number of variants of different crates / bottles due to a high degree of individualization of the products and with smaller retailers, it is increasingly not possible to always provide a pallet of one type. As a result, the error rate will continue to increase.

This fact represents the greatest challenge for beverage manufacturers nowadays.

In other sectors, too, such as the automotive industry / supplier industry, empties (load carriers, KLTs) are often used. These load carriers are used to hold special components that are used, for example, in automobile production. These load carriers are usually made to accommodate very specific items and are therefore located in a circulatory system between the automobile manufacturer and its suppliers.

These load carriers have a high value because they are specially built for a product.

For the supplier companies, this means that they have to manage the incoming goods of the empties just like the beverage manufacturers. Regardless of whether the load carriers belong to the supplier or the automobile manufacturer. If they don't, there is a risk that they will have to bear the high costs of missing load carriers. Here, too, it is necessary to manually record the empties (i.e. when the load carrier is empty, it has a barcode label when it is full). A quick and reliable detection of the empties by means of the presented invention also makes sense in industries that work with individual load carriers for holding components.

Proceeding from this, the invention is based on the object of providing a dynamic, mobile and real-time capable quantity and type detection of goods during unloading or loading which reliably detects incorrect sorting.

According to a first aspect of the present invention, this object is achieved by a forklift truck according to claim 1. The forklift truck comprises: first camera system with a first camera which is designed to capture a first side of goods to be picked up; a second camera system with a second camera which is designed to capture a second side of the goods during or after they have been picked up by the forklift truck, the first side differing from the second side; a first sensor system with a first sensor which is designed to detect a distance between the forklift truck and the goods; a controller which is designed to output a first activation signal to the first camera when the distance detected by the first sensor falls below a minimum distance; and a transmission unit which is designed to transmit the image data of the goods captured by the first camera and by the second camera in each case to a data processing system for identifying the goods. With the first and the second camera of the forklift truck, the image data of the goods to be picked up / picked up, in particular a pallet of empty boxes, are generated in real time and transmitted to the data processing system. Analysis of the Image data has the advantage that the goods, in particular a type detection of the empties, can be determined. This allows an exact recording of the current stocks of empties and thus a more efficient production process.

According to a further aspect of the present invention, the first sensor system is based on ultrasound or radar in order to determine the distance from the forklift exactly.

According to yet another aspect of the present invention, the forklift truck comprises a second sensor system with a second sensor which is designed to detect that the forklift truck has picked up the goods.

According to yet another aspect of the present invention, the controller is also designed to output a second activation signal to the second camera when it is determined by the second sensor that goods have been picked up.

According to yet another aspect of the present invention, the second camera system has an extendable device which is designed to bring the second camera into a position which enables the second side of the goods to be detected. A rear side of the goods W can thus be detected.

According to yet another aspect of the present invention, the first camera and the second camera are 3D cameras. In this way, the dimensions of the goods, such as height and width, can be determined.

According to yet another aspect of the present invention, the forklift includes the data processing system. This has the advantage that the image analysis can be carried out on the forklift in real time, and the information about the identified goods can be made available directly to an inventory control system.

According to yet another aspect of the present invention, the first camera system further comprises a first mirror and a second mirror, each with a device which is designed to position the mirrors so that they have a third side and a fourth side of the goods picked up by the forklift truck capture. In this way, other sides of the goods can be recorded, which enables more precise identification of empties.

According to yet another aspect of the present invention, the first camera is attached to a mast of the forklift.

According to yet another aspect of the present invention, the second camera system is arranged in one of the forks of the forklift truck. This has the advantage that the second camera can capture a lower side of the goods while the goods are being picked up. So For example, it can be ensured that no empty crates are missing from a pallet

According to yet another aspect of the present invention, after the goods have been picked up by the extendable device, the second camera is positioned in such a way that it captures the second side of the goods, the second side of the goods corresponding to the rear side facing away from the forklift truck. The retractable device enables the second camera to be brought into different positions in order to ensure complete coverage of the rear side of the goods.

According to yet another aspect of the present invention, the first mirror and the second mirror are each attached to the forklift and are positioned after the goods have been picked up by the respective device so that the third and fourth sides of the goods, which correspond to the lateral sides, reflect the first mirror and the second mirror being sighted by the first camera after the goods have been picked up. The mirrors are attached to the side of the forklift and can be extended / unfolded using the respective device. The first camera can then also capture the image data of the side part of the goods by means of the extended / unfolded mirror, which improves the goods identification.

According to yet another aspect of the present invention, the data processing system in the forklift is designed to carry out preprocessing of image data that are captured by one or more camera systems of the forklift, the preprocessing filtering sensor-related errors, preparing the data format of the image data, and sensor data fusion , feature extraction and machine segmentation.

According to yet another aspect of the present invention, the data processing system is further designed to carry out an identification of the goods, in particular of empty crate types, and a number of empty crates, by means of one or more of the following classifications from the preprocessed image data: a geometrical classification based on geometrical data that are determined from the image data, or their extracted features, a texture-based classification based on color data, which are determined from the image data, and their extracted features. Furthermore, the bottle types of the goods W can also be determined.

According to yet another aspect of the present invention, the data processing system is further designed to carry out a statistical estimation and an estimation by machine learning on the basis of the captured image data in order to predict the occupancy of individual empty boxes of the goods picked up determine. In this way, specific image processing algorithms and machine learning methods (such as "K-Nearest Neighbors", neural networks, etc.) can be combined with one another in order to ensure reliable detection.

According to yet another aspect of the present invention, a system according to claim 13 comprises a forklift truck with a positioning system which is designed to detect a geographical position of the forklift truck in a warehouse in real time; a first camera system with a first camera which is designed to capture a side of a goods to be picked up facing the forklift truck; a first sensor system with a first sensor which is designed to detect a distance between the forklift truck and goods to be picked up; a controller which is designed to output a first activation signal to the first camera when the distance detected by the first sensor falls below a minimum distance; a transmission system which is designed to transmit the image data of the goods captured by the first camera to a data processing system; a second camera system with a second camera, which is arranged in a warehouse, and which is designed to capture at least one side of the goods facing away from the forklift truck during and after the goods are picked up by the forklift truck, the data processing system being designed by the the first camera and the second camera to analyze captured image data in order to thereby identify the goods. This makes it possible to merge two concepts, i.e. an external and a frontal image acquisition of a product. By combining the captured image data from the first camera of the forklift truck with the image data from the second camera of the system, the goods, in particular types of empty containers, can be identified.

According to yet another aspect of the present invention, the second camera of the second camera system is attached in an area of a goods unloading point within the warehouse, preferably on the hall ceiling; or attached to a gate of the warehouse, which the forklift truck passes with the goods picked up.

According to yet another aspect of the present invention, the data processing system is provided by the forklift itself.

According to yet another aspect of the present invention, the positioning system of the forklift is also designed to provide the image data captured by the first camera with the geographical position in order to subsequently assign the image data captured by the first camera to the image data captured by the second camera enable, the image data captured by the first camera and the second camera each additionally having a time stamp.

According to yet another aspect of the present invention, the data processing system is designed to identify the forklift truck by analyzing the image data captured by the second camera in order to enable the image data captured by the first camera to be associated with the image data captured by the second camera.

According to yet another aspect of the present invention, the data processing system is designed to carry out preprocessing on the basis of the associated image data of the first camera and the second camera, the preprocessing filtering of sensor-related errors, preparing the data format of the image data, and sensor data fusion , feature extraction, and machine segmentation.

According to yet another aspect of the present invention, the data processing system is further designed to carry out an identification of the goods, in particular of empty crate types and a number of empty crates, from the preprocessed image data using one or more of the following classifications: (a) a geometric classification based on geometric data determined from the image data or their extracted features, and (b) texture-based classification based on color data determined from the image data and their extracted features.

According to yet another aspect of the present invention, the second camera system comprises a plurality of mirrors which are arranged in the warehouse and which are designed to capture the sides of the goods facing away from the forklift truck, the first camera also being designed to display the mirrors at or after Aiming at picking up the goods.

According to yet another aspect of the present invention, the second camera system is formed by a drone with a camera.

• Fig. 1A stellt eine perspektivische Ansicht dar, die ein System mit einem Gabelstapler gemäß ersten Ausführungsformen zeigt.
• Fig. 1B stellt eine perspektivische Ansicht dar, die den Gabelstapler gemäß der ersten Ausführungsformen im Detail zeigt.
• Fig. 2A stellt einen perspektivischen Ausschnitt dar, der eine Gabel des Gabelstaplers gemäß der ersten Ausführungsformen vor einer Warenaufnahme zeigt.
• Fig. 2B stellt einen perspektivischen Ausschnitt dar, der die Gabel des Gabelstaplers gemäß der ersten Ausführungsformen während einer Warenaufnahme zeigt.
• Fig. 2C stellt einen perspektivischen Ausschnitt dar, der die Gabel des Gabelstaplers gemäß der ersten Ausführungsformen nach einer Warenaufnahme zeigt.
• Fig. 3A stellt eine perspektivische Seitenansicht dar, die den Gabelstapler gemäß der ersten Ausführungsformen mit einem ersten Spiegel in einer ersten Position zeigt.
• Fig. 3B stellt eine perspektivische Seitenansicht dar, die den Gabelstapler gemäß der ersten Ausführungsformen mit dem ersten Spiegel in einer zweiten Position zeigt.
• Fig. 3C stellt eine perspektivische Draufsicht dar, die den Gabelstapler gemäß der ersten Ausführungsformen mit dem ersten Spiegel und einem zweiten Spiegel in der zweiten Position zeigt.
• Fig. 4 stellt ein Beispiel der Bilddatenverarbeitung dar, das eine Segmentierung eines erfassten Bildes zeigt.

Further advantages and details of the invention are explained in more detail with reference to the exemplary embodiments shown in the schematic figures.

• Figure 1A Fig. 13 is a perspective view showing a system including a forklift according to first embodiments.
• Figure 1B Fig. 13 is a perspective view showing the forklift truck according to the first embodiment in detail.
• Figure 2A shows a perspective cutout showing a fork of the forklift according to the first embodiment in front of a goods pick-up.
• Figure 2B represents a perspective detail showing the fork of the forklift according to the first embodiment during a goods pick-up.
• Figure 2C represents a perspective detail showing the fork of the forklift according to the first embodiment after receiving goods.
• Figure 3A Fig. 10 is a side perspective view showing the forklift truck according to the first embodiment with a first mirror in a first position.
• Figure 3B Figure 10 is a side perspective view showing the forklift truck according to the first embodiment with the first mirror in a second position.
• Figure 3C Fig. 10 is a top perspective view showing the forklift according to the first embodiment with the first mirror and a second mirror in the second position.
• Fig. 4 FIG. 10 illustrates an example of image data processing showing segmentation of a captured image.

Detailed description

In the following with reference to Figure 1A and Figure 1B a system 100 with a forklift 10 according to first embodiments is described in more detail. The in Figure 1A and Figure 1B The forklift 10 shown comprises a first camera 12 which detects a first side S1 of a goods W to be picked up, such as a pallet of empty boxes. The field of view of the first camera 12 is in Figure 1A indicated by a dashed line. In the in Figure 1A and Figure 1B In the example shown, the first camera 12 is attached to a lifting mast 11 of the forklift 10. In a further (not shown) example of the first embodiment, the first camera 12 can also be attached to a load carrier 17 of the forklift 10.

The forklift truck 10 further comprises a first sensor 16 which determines a distance between the forklift truck 10 and the goods W. For example, the first sensor 16 can determine the distance to the goods W by measuring the transit time of electromagnetic waves or sound waves, which is indicated by the dashed line in FIG Figure 1A is marked. The first sensor 16 comprises an ultrasonic sensor, a radar sensor and a LIDAR (“Light Detection and Ranging”) sensor. In the in Figure 1A and Figure 1B The example shown, the first sensor 16 is attached to the roof of the forklift 10. In further (not shown) examples of the first embodiment, the first sensor 16 can also be attached to a lifting mast 11 of the forklift 10, or can be attached to one of the forks 13 or to the load carrier 17 of the forklift.

A controller 20 of the first embodiments comprised by the forklift 10 as shown in FIG Figure 1B As shown, if the distance determined by the first sensor 16 falls below a minimum distance, a first activation signal can be output to the first camera 12. The first camera 12 is activated by the first activation signal and begins to record image data of the first side S1 of the goods W. For example, the first camera records a front side of the goods W to be picked up facing the forklift truck 10.

In addition to the first camera 12, the forklift truck 10 of the first embodiment comprises a second camera 14 which, during or after the recording, detects further sides of the goods W that differ from the first side S1. In the in Figure 1A and Figure 1B In the example shown in the first embodiment, the second camera 14 is received in one of the forks 13 of the forklift 10. In a further example (not shown) of the first embodiment, the second camera 14 can also be attached to the lifting mast 11 of the forklift 10.

The forklift truck 10 of the first embodiment can preferably comprise a second sensor 18 which determines the pick-up of the goods W by the forklift truck 10. For example, the in Figure 1B The second sensor 18 shown is an ultrasonic sensor. The second sensor 18 can comprise further sensors, for example a load sensor, which detects the picking up of the goods W by means of a load indication. In the in Figure 1B In the example shown, the second sensor 18 is received in one of the forks 13 of the forklift 10. In a further example (not shown) of the first embodiment, the second sensor 18 can also be attached to the lifting mast 11 or to the load carrier 17 of the forklift 10.

Preferably, when the second sensor 18 determines that the goods W have been picked up, the controller 20 of the forklift 10 can output a second activation signal to the second camera 14 of the forklift 10 in order to activate the second camera 14. For example, the second camera 14 detects a lower side of the goods W while it is being picked up by the forks 13 of the forklift 10. In a further example, the second camera records a rear side of the goods W facing away from the forklift truck 10 after it has been recorded. This will be referred to later on Figures 2A to 2C described in more detail.

Furthermore, the movement of the forklift 10 in the area of a goods unloading point within the warehouse offers the possibility of capturing image data of the sides of the goods W facing away from the forklift 10 with a small number of external cameras 114. Therefore, in Figure 1A The system 100 of the first embodiment shown comprise one or more external cameras 114, which are arranged in a warehouse, and which Detect at least one side of the goods W facing away from the forklift truck 10 during and after the goods W are picked up by the forklift truck 10.

An external camera 114 can preferably be attached within a hall ceiling included in the warehouse in order to capture an upper side of the goods W. In a further example (not shown), an external camera 114 can be attached to a lamp post in the warehouse in order to capture a lateral side of the goods W. In yet another example (not shown), an external camera 114 can be attached to a passage gate of the warehouse through which the forklift truck with the picked up goods W passes, in order to capture a lateral side of the goods W.

An external camera 114 can preferably be recorded in an airworthy drone, which films the forklift 10 during and after the goods W are picked up. For example, when the controller outputs the first activation signal, a signal with the current geographical coordinates of the forklift 10 can also be output to the drone. The drone then captures the remaining sides of the goods W.

In a further example according to the first embodiment, an external camera 114 can receive the first or the second activation signal that is output by the controller 20 of the forklift 10 and begin to transmit the image data captured by the external camera 114 to a data processing system 24.

The number of cameras and sensors that the forklift 10 and system 100 may include can vary. In an example of second embodiments (not shown), the forklift 10 can only include one camera and the system 100 can only include one external camera. In a further example (not shown) of the second embodiment, the forklift 10 can include only one sensor and the system 100 can include one or more sensors.

The cameras 12, 16 of the forklift truck 10 and the external cameras 114 of the system 100 are preferably designed with lighting options so that the cameras can achieve correct image data acquisition even in dark surroundings through sufficient lighting.

The cameras are preferably 3D cameras that operate on the principle of the light flow method in order to additionally enable a determination of the depth data of the goods W to be recorded or recorded. The cameras can also include network cameras that provide digital signals at the output in the form of a video stream that can be transmitted via Internet Protocol (IP).

Furthermore, the forklift 10 comprises a transfer unit 22, as in FIG Figure 1B which can transmit the image data captured by the first 12 and the second camera 14 of the forklift 10 to a data processing system 24 for the identification of the goods W.

In addition, the forklift 10 can include a location system (not shown) that detects a geographic position of the forklift in the warehouse in real time. For example, the positioning system can comprise a GPS sensor, which makes it possible to determine the geographic coordinates of the forklift truck 10 in real time. The coordinates determined in this way can be transmitted together with the captured image data of the forklift 10 and a time stamp by the transmission unit 22 to the data processing system 24. The data processing system 24 can then, based on the transmitted coordinates and the time stamp, combine the image data captured by the forklift truck 10 with the image data captured by an external camera 114 of the system 100.

Alternatively, or in addition to the location system of the forklift truck 10, the forklift truck 10 can be identified by analyzing the image data of an external camera 114 of the system 100, for example via a QR code, in order to enable the image data to be assigned to the image data of the forklift truck 10.

With reference to Figures 2A to 2C the acquisition of the image data by the second camera 14 of the forklift 10 according to first embodiments is described in more detail. Figure 2A shows a perspective detail of the forklift 10 according to the first embodiment, which shows the fork 13 with the second camera 14 before the goods W are picked up. The second camera 14 can preferably be formed with impact-resistant and / or waterproof materials, which protect the second camera 14 from impacts when picking up goods and / or from moisture. For example, the second camera 14 can be arranged in a shock-proof and / or waterproof housing that is received in the fork 13 of the forklift.

Figure 2B shows a perspective detail of the forklift 10 according to the first embodiment, which shows the fork 13 with the second camera 14 while the goods W are picked up. The second camera can preferably comprise a 3D camera which uses a time of flight (TOF) method to determine the dimensions of the goods W during their recording, as in FIG Figure 2B shown, determined. For example, it can be ensured that no empty crates are missing from a pallet.

Figure 2C shows a perspective section of the forklift 10 according to the first Embodiments represent the fork 13 with the second camera 14 after the recording of the Goods W shows. Preferably, the second camera can be connected to an extendable device 30 (not shown). The extendable device comprises a telescopic device. This makes it possible to position the second camera 18 in such a way that the rear side of the goods W facing away from the forklift truck can be detected.

In a further example of the first embodiment, the second camera can be attached to the lifting mast 11 of the forklift (not shown). In this case, the second camera 18 can be connected to the extendable device 30 so that the second camera 18 is positioned in such a way that it detects an upper side of the goods W or a lateral side of the goods W.

As an alternative or in addition to the cameras comprised by the forklift truck 10, the forklift truck 10 can include one or more mirrors.

With reference to Figures 3A to 3C the acquisition of the image data according to the first embodiment with a first mirror 26 and a second mirror 28 which are attached to the forklift 10 will be described in more detail. Figure 3A Figure 12 illustrates a perspective view of the forklift 10 showing the first mirror 26 in a first position. Figure 3B Figure 10 illustrates a perspective view of the forklift 10 showing the first mirror 26 in a second position. Figure 3C Figure 10 illustrates a top perspective view of forklift 10 showing first mirror 26 and second mirror 28 in the second position.

In Figure 3A For example, the first mirror 26, which is connected to an extendible / fold-out device, is attached to the side of the forklift 10 and is in the first position, i.e. in a retracted / folded position. Likewise, the second mirror 28 is attached on the opposite side of the forklift 10 (not shown). After the goods W have been picked up by the forklift truck 10, the first mirror 26 and the second mirror 28 can be brought into the second position, that is to say extended / folded out, in order to reflect the lateral sides S3, S4 of the goods W. This is in Figure 3B shown. The first mirror 26 and the second mirror 28 can preferably be extended when it is determined by the second sensor 18 that the goods W are being picked up.

The first camera 12 can preferably be rotated up to 90 ° in two directions in order to vary the field of view of the first camera 12. This enables the first mirror 26 and the second mirror 28 to be aimed at, as in FIG Figure 3C to capture the image data of the side sides of the goods W.

In a further example according to the first embodiment, the system 100 can also comprise one or more mirrors. For example, can, alternatively, or in addition to the external cameras 114, one or more mirrors can be fitted in the warehouse in the area of the goods unloading point (not shown) in order to reflect at least one side of the goods W facing away from the forklift truck 10. In one example, the first camera 10 of the forklift 10 can then aim at the mirror in the warehouse to capture the image data.

The image data processing is explained in detail below. After the image data of the first camera 12 and the second camera 18 of the forklift 10, as well as the external cameras 114 of the system 10, have been captured and transmitted to the data processing system 24, data processing takes place.

The forklift truck 10 preferably contains the data processing system 24. The data processing system 24 can also be contained in an external processing unit, for example a central computer.

With the coordinates of the forklift 10 in the warehouse determined by the location system of the forklift 10, the image data captured by the forklift 10 can be combined with those of the image data captured by the external cameras 114. In a first step, the image data is preprocessed.

In the preprocessing, sensor-related errors such as noise, overexposure, etc. are preferably filtered out of the image data by machine. The data format is then prepared, for example by converting the image data to gray levels or by transforming the depth data of the goods W to determine the dimensions of the goods W. The data determined by the sensors of the forklift truck 10 and the system 100 are preferably also merged with the image data .

A machine segmentation of the captured image data is preferably carried out, as in FIG Fig. 4 shown. Many methods of automatic segmentation are possible. Basically, they are often divided into pixel, edge and region-oriented processes. In addition, a distinction is made between model-based methods, in which a certain shape of the objects is assumed, and texture-based methods, in which an internal, homogeneous structure of the objects is also taken into account. Different methods can also be combined to achieve better results.

Features of the captured image data can preferably be extracted. Examples include, but are not limited to, SIFT ("Scale-Invariant Feature Transform"), HOG ("Histogram of Oriented Gradients"), and SURF (" Speeded Up Robust Features").

SIFT is an algorithm for the detection and description of local features in images. The detector and the feature descriptions are, within certain limits, invariant compared to coordinate transformations such as translation, rotation and scaling. They are also robust against lighting variations, image noise and lower geometric deformations of a higher order, such as those caused by projective imaging of an object from different points of view in space.

Similar to SIFT, HOG is an algorithm for feature recognition in images that shows the appearance and shape of objects within an image even without detailed knowledge of the positions of edges or corners by distributing the local intensity or the arrangement of the edges. For this purpose, the image is broken down into sub-areas and the orientations of all edges are determined for each sub-area and their number is saved as a histogram.

The SURF algorithm is based on the same principles and steps as SIFT. However, the details in each step are different. The algorithm consists of three main parts: identification of points of interest, description of the local environment and mapping.

In a second step, based on one of the procedures described below, the goods W, in particular the empty packaging types of the pallet (s) picked up, are identified.

A geometric classification by means of (possibly segmented) depth data or their extracted features, such as height, depth, width and number of boxes, which are obtained from the preprocessed image data, is made possible by comparison with data from a previously created database. This allows the types of empties to be identified, in particular the types of crates of the goods W.

A texture-based classification using (possibly segmented) color data or their extracted features includes the application and evaluation of statistical methods for computer-aided recognition of patterns in images. By statistical evaluation of the preprocessed image data (for example by means of the median color tone determination) and the use of machine learning (for example "shallow learning", "deep learning"), an algorithm can be trained with sufficiently real data or their features. This trained algorithm can then be validated and verified using real data and then used to identify empties.

The identification of the empty packaging types preferably comprises a combination of the geometric and texture-based classification with merged sensor data.

A prediction of the most likely occupancy of the individual empties crates is preferably made by analyzing all visible sides of the goods W, for example two per Pallet. This is done using a statistical estimate and / or an estimate using machine learning methods.

In the last step, the corresponding results of the image data processing are then sent to an ERP system via a standardized communication path (for example using a network protocol such as the "Simple Object Access Protocol" (SOAP)) and standardized data formats (for example CSV, XML, JSON-RPC) transfer. A production planning and control system PPS can thus work with empties data in a high quality.

Various embodiments have been described in detail above. It is clear to a person skilled in the art that various special features of special embodiments can be combined with features of other respective embodiments.

Claims (15)

1. A forklift truck (10), comprising:

   a first camera system with a first camera (12) configured to capture a first side (S1) of goods (W) to be picked up;

   a second camera system with a second camera (14) configured to capture a second side of the goods while or after picking them up with the forklift, the first side being different from the second side;

   characterized by

   a first sensor system with a first sensor (16) configured to detect a distance of the forklift truck from the goods;

   a controller (20) configured to output a first enabling signal to the first camera when the distance detected by the first sensor is less than a minimum distance; and

   a transmission unit (22) configured to transmit the image data of the goods captured by the first camera and by the second camera, respectively, to a data processing system (24) for identifying the goods.
2. The forklift truck according to claim 1, wherein the first sensor system is an ultrasonic or radar system.
3. The forklift truck according to any one of claims 1 to 2, further comprising:
   a second sensor system with a second sensor (18) configured to capture the forklift truck while picking up the goods.
4. The forklift truck according to claim 3, wherein the controller is further configured to output a second enabling signal to the second camera when it is determined by the second sensor that goods have been picked up.
5. The forklift truck according to any one of claims 1 to 4, wherein the second camera system includes an extendible device (30) configured to move the second camera to a position enabling capturing of the second side of the goods.
6. The forklift truck according to any one of claims 1 to 5, wherein the first camera and the second camera are 3D cameras capable of detecting the dimensions of the goods and/or wherein the forklift truck includes the data processing system.
7. The forklift truck according to any one of claims 1 to 6, wherein the first camera system further comprises a first mirror (26) and a second mirror (28) each having a device (32) configured to position the mirrors for capturing a third side (S3) and a fourth side (S4) of the goods picked up by the forklift truck.
8. The forklift truck according to any one of claims 1 to 7, wherein the first camera is mounted to a mast (11) of the forklift truck, or
   wherein the second camera system is arranged in one of the forks (13, 15) of the forklift truck and wherein the second camera captures the lower side of the goods while the goods are picked up.
9. The forklift truck according to claim 5 and claim 8, wherein the second camera is positioned by the extendible device so as to capture the second side of the goods after the goods have been picked up and wherein the second side of the goods corresponds to the rear side facing away from the forklift truck.
10. The forklift truck according to any one of claims 1 to 6, 8 or 9 and according to claim 7,
    wherein the first mirror and the second mirror are respectively mounted to the forklift truck and are positioned by the respective device after the goods have been picked up such that the third and fourth sides of the goods corresponding to the lateral sides are reflected, wherein the first mirror and the second mirror are aimed at by the first camera after the goods have been picked up.
11. The forklift truck according to any one of claims 1 to 10 having a data processing system, wherein the data processing system is configured to perform preprocessing of image data captured by one or more camera systems of the forklift truck, wherein the preprocessing comprises filtering out of sensor-related errors, preparing the data format of the image data, merging of the sensor data, extracting features, and machine segmentation, wherein preferably the data processing system is further configured to perform an identification of the goods, in particular of types of crates with empty bottles as well as of a number of the crates with empty bottles, using the preprocessed image data by means of one or more of the following classifications:

    (a) a geometric classification using geometric data determined from the image data or their extracted features,

    (b) a texture-based classification using colour data determined from the image data or their extracted features.
12. The forklift truck according to claim 11, wherein the data processing system is further configured to perform a statistical estimation and an estimation by means of machine learning based on the captured image data to determine a prediction of occupancy at individual crates with empty bottles of the picked up goods.
13. A system (100) comprising,

    a forklift truck (110) having:

    a positioning system configured to detect a geographical position of the forklift truck in a warehouse in real time;

    a first camera system with a first camera (112) configured to capture a side of goods (W) to be picked up facing the forklift truck;

    a first sensor system with a first sensor configured to detect a distance of the forklift truck from goods to be picked up;

    a controller configured to output a first enabling signal to the first camera when the distance detected by the first sensor is less than a minimum distance;

    a transmission system configured to transmit the image data of the goods captured by the first camera to a data processing system;

    a second camera system with a second camera (114) located in a warehouse and configured to capture at least one side of the goods facing away from the forklift truck during and after picking up the goods with the forklift truck,

    wherein the data processing system is configured to analyze the image data captured by the first camera and the second camera to thereby identify the goods.
14. The system according to claim 13, wherein

    the second camera of the second camera system is installed in an area for unloading goods within the warehouse, preferably on the ceiling; or

    the second camera of the second camera system is installed at a passage gate of the warehouse which the forklift truck with the picked-up goods passes.
15. The system according to any one of claims 13 or 14, wherein the data processing system is provided by the forklift truck itself, or

    wherein the positioning system of the forklift truck is further configured to provide the image data captured by the first camera with the geographical position, thereby enabling a subsequent association of the image data captured by the first camera with the image data captured by the second camera, wherein the image data captured by the first camera and the image data captured by the second camera each additionally comprise a time stamp, or

    wherein the data processing system is configured to identify the forklift truck by analyzing the image data captured by the second camera to enable association of the image data captured by the first camera with the image data captured by the second camera, or

    wherein the data processing system is configured to perform preprocessing on the basis of the associated image data of the first camera and the second camera, the preprocessing comprising filtering out sensor-related errors, preparing the data format of the image data, merging of sensor data, extracting features and machine segmentation, and/or

    wherein the data processing system is further configured to perform an identification of the goods, in particular of types of crates with empty bottles as well as of a number of the crates with empty bottles, using the preprocessed image data by means of one or more of the following classifications:

    (a) a geometric classification using geometric data determined from the image data or their extracted features, and

    (b) a texture-based classification using colour data determined from the image data or their extracted features or

    wherein the second camera system comprises a plurality of mirrors located in the warehouse and configured to capture the sides of the goods facing away from the forklift truck, the first camera being further configured to be aimed at the mirrors while or after picking up the goods, or

    wherein the second camera system is constituted by a drone with a camera.

Images