IT202300007002A1 - LOCALIZATION SYSTEM FOR INDUSTRIAL VEHICLES - Google Patents

Description

Description of a Patent for an Industrial Invention entitled:

?LOCALIZATION SYSTEM FOR INDUSTRIAL VEHICLES?.

DESCRIPTION

The present invention relates to a localization system for industrial vehicles.

As is known, the localization of industrial vehicles is a problem of fundamental importance for the management of fleets and for the traceability of movements within industrial plants.

In fact, vehicles such as forklifts, shuttles and carts are used to move things or people both in open spaces and inside buildings.

The most reliable solution currently available for knowing the position of a vehicle in real time is to mount a Global Navigation Satellite System (GNSS) device on board, which intercepts electromagnetic signals from one or more satellite constellations and provides the position of the vehicle in geographic coordinates.

However, the electromagnetic radiation of satellites is severely degraded or even absent inside buildings, making the use of this system ineffective.

To solve the problem of indoor vehicle localization, several solutions have been introduced, each based on a specific reference technology.

For example, the use of UltraWideBand (UWB) antennas installed on board the vehicle and deployed inside the building is known to allow the operation of a GNSS to be replicated in closed environments.

Alternatively, the use of Light Imaging, Detection, And Ranging (LIDAR) devices together with the installation of several highly reflective reference points in suitable positions is known.

This way, it is possible to determine the position of the vehicle using the distances of the reflectors detected by the LIDAR device.

Finally, several solutions have been developed to determine the position of vehicles with the help of cameras.

For example, there are known solutions that use a camera mounted on top of the vehicle and facing upwards so as to observe the ceiling of the plant, on which appropriate markers, such as labels or similar, are placed.

Furthermore, there are known solutions that use a camera installed on board the vehicle and facing the floor. Also in this case, appropriate markers recognizable by the system are installed on the ground.

However, in all known solutions described above, the installation and maintenance of an adequate reference point infrastructure is necessary, since the position of the vehicle is determined in relation to these references.

Necessarily, this requires significant effort on the part of the user of the system.

In fact, at the time of system startup, the deployment of a high number of devices, both active and passive, is required, while during the life cycle their constant maintenance is necessary so that the system does not degrade its performance.

Furthermore, under certain conditions the installation of such devices is prohibited or made problematic by the particular configurations of the environment.

The main task of the present invention is to devise a localization system for industrial vehicles that allows to overcome the limitations exposed above for known solutions, allowing the localization of a vehicle through natural references present within the environment, without requiring either the installation of references at start-up or their subsequent maintenance.

The above-mentioned purposes are achieved by the present localization system for industrial vehicles according to the characteristics described in claim 1.

Other features and advantages of the present invention will become more evident from the description of a preferred, but not exclusive, embodiment of a localization system for industrial vehicles, illustrated by way of example, but not of limitation, in the attached drawings in which:

Figure 1 and Figure 2 illustrate a possible industrial vehicle that can be used with the localization system according to the invention;

Figure 3 illustrates a possible industrial environment within which the management of industrial vehicles and materials is carried out using the localization system according to the invention;

Figure 4 is a block diagram schematically illustrating the localization system according to the invention.

With particular reference to these figures, the term 1 generally refers to a localization system for industrial vehicles, which can be used in particular to manage the movement of materials within an industrial area, such as a warehouse or similar.

The localization system 1 comprises acquisition means 2 installed on at least one industrial vehicle V to be located and configured to acquire at least one image I of the environment surrounding the vehicle V and/or data relating to the distances of objects with respect to the vehicle V.

The acquisition means 2 are appropriately positioned so as to observe the environment surrounding vehicle V.

According to preferred embodiments, the acquisition means 2 comprise at least one of a stereoscopic camera or a LIDAR system.

The localization system 1 further comprises at least one storage unit 3 comprising at least one global map GM of an industrial area A within which the vehicle V must move.

Furthermore, the localization system 1 comprises at least one processing unit 4 operatively connected to the acquisition means 2 and to the storage unit 3 and configured to determine the position of the vehicle V within the area A starting from the acquired image I and/or from the acquired distance data.

With reference to a possible embodiment, illustrated in the figures, the processing unit 4 is implemented by means of a local processor 5 installed on the vehicle V, while the storage unit 3 is implemented on a remote processor 6. In this case, the local processor 5 can communicate with the remote processor 6 by means of wireless communication (WiFi, Bluetooth or similar).

However, alternative embodiments are not excluded in which the processing unit 4 and the storage unit 3 are implemented and located differently.

For example, both the processing unit 4 and the storage unit 3 can be implemented on a remote computer, while the vehicle V in this case can be equipped with wireless communication means for sending the images I and/or the distance data collected by the acquisition means 2 to the remote computer.

Alternatively, both the processing unit 4 and the storage unit 3 may be installed on the vehicle V.

Usefully, the localization system 1 may include an on-board computer 7 installed on the vehicle V and configured to interface with the vehicle user.

Advantageously, according to the invention, the global map GM comprises global coordinates relative to a plurality of global references GR, wherein the global references GR correspond to fixed structural elements E present within the area A and to material M deposited and immobile within the area A, and wherein the global coordinates are determined with respect to a predefined origin point O on the global map GM,

Additionally, the processing unit 4 is configured to iteratively repeat the following steps:

- starting from the at least one acquired image I and/or from the acquired distance data, determine a plurality of local coordinates relative to a plurality of local references LR, where the local references LR correspond to fixed structural elements E present within the area A and/or to material M deposited and immobile within the area A, and where the local coordinates are determined considering a point of origin on the vehicle V (step 41 of figure 4);

- locate global GR references on the global GM map corresponding to at least some of the determined local LR references (step 42);

- starting from the identified global references GR, determine the position P of the vehicle V within the area A by transforming the local coordinates of the vehicle V into corresponding global coordinates within the global map GM (step 43);

- once the position P of vehicle V has been determined, if at least one of the determined local LR references does not correspond to any global GR reference on the global GM map and/or if on the global GM map there is at least one global GR reference that is not present among the determined local LR references, update the global GM map by adding/deleting at least one global GR reference (step 44). In particular, if at least one of the determined local LR references does not correspond to any global GR reference on the global GM map, then the processing unit 4 adds such global GR reference on the global GM map.

If, however, there is at least one global reference GR on the global GM map that is not present among the determined local LR references, then the processing unit 4 eliminates this global reference GR from the global GM map.

As regards the transformation of the local coordinates of the vehicle V into corresponding global coordinates within the global map GM, according to a possible and preferred form of implementation, given the correspondences between local and global references, the translations of the centroids, or barycenters, of the local points and of the global points in the origins of the respective coordinates are first considered. Subsequently, the Kabsch algorithm is applied to estimate the rotation. The final transformation is obtained by composing the previous transformations.

By means of the localization procedure described above, the localization system 1 is able to know in real time the position of each vehicle V within the mapped industrial area A and, therefore, can externally transmit this position P to other management systems, for example, to track the movements of material M within this area and evaluate various warehouse performance indices (step 45).

With reference to the above, it is specified that within this description the adjective "local" refers to the coordinates that describe the reference points observed by a vehicle V with respect to itself.

The term ?global? instead indicates the coordinates of the elements of the mapped industrial area A, decided at the time of initialization of the localization system 1 and choosing the origin of the axes.

Therefore, to know the position P of the vehicle V it is necessary to determine the transformation that maps the local coordinates to the global coordinates and vice versa.

By extension, the term "local map" indicates the set of references observed by vehicle V and expressed in local coordinates, while the term "global map" refers to the set of global coordinates that represent the most recent version of all known references of the plant.

It should also be noted that the localization system 1 records only stored and immobile M material. In this way, it is possible to build a solid and coherent global map GM of references that can be used for vehicle localization.

Consequently, the images I of the surrounding environment acquired by each of the localized vehicles V allow the modification of the global map GM by adding or removing references relating to deposited and/or retrieved material M. In this way, a dynamic management of the global map GM of references is possible. Conversely, an updated global map GM of material M allows the correct localization of the vehicles V within the mapped industrial area A.

According to a preferred embodiment of the localization system 1, the global map update criterion is based on two fundamental hypotheses:

- vehicles V move while material M is stationary and

- the material M can be collected or deposited using a vehicle V appropriately equipped with the system, but not otherwise modified or deformed.

In particular, the second hypothesis specifies that the localization system 1 describes the global map GM of the global references GR pre and post modifications, but does not manage any transition phase between the two. In this way the global map GM of the global references GR contains only existing and valid material M for the subsequent localizations of the vehicles V. Furthermore, the processing unit 4 is configured to determine the orientation D of the vehicle V starting from the position P of the vehicle itself and from the global coordinates of the identified global references GR.

The localization system 1 uses the position information P and orientation D of the vehicle V to compare the environment observed by the acquisition means 2 and that saved in the global map GM.

In fact, knowing the position P and orientation D of the vehicle V, it is possible to transpose the observed reference points into global coordinates and compare them with the known ones.

Advantageously, according to a preferred embodiment, the localization system 1 comprises at least one presence sensor 8 installed on the gripping means F of the material M of the vehicle V. The presence sensor 8 is operatively connected to the processing unit 4 and is configured to detect the presence/absence of material M on the gripping means F.

With reference to this preferred embodiment, the processing unit 4 is configured to perform the following steps:

- detect using presence sensor 8 a loading/unloading operation of material M onto/from vehicle V (step 46);

- record the P position of vehicle V during this unloading/loading operation (step 47);

- starting from the recorded vehicle position P, determine a material unloading/loading position M (step 48);

- at this unload/load position, generate an invalidation volume VI within the global GM map (step 49).

Still with reference to the preferred embodiment of the localization system 1, the step 44 of updating the global map, performed by the processing unit 4, comprises the following steps:

- if the GR global reference to be added/removed from the GM global map matches a generated VI invalidation volume, then processing unit 4 adds/removes the GR global reference to/from the GM global map;

- if the GR global reference to be added/removed from the GM global map does not match a generated VI invalidation volume, then the processing unit does not add/delete the GR global reference to/from the GM global map.

Therefore, whenever the presence sensors 7 on the gripping means F detect a loading or unloading operation, the localization system 1 records the exact position P of the vehicle, determined by the localization algorithm. The position of the gripping means F at the time of loading or unloading allows determining an invalidation volume VI in global coordinates. This volume is invalidated by the logistic operation of the vehicle V. Only the previously invalidated volumes admit modification by means of the global map update step GM.

In particular, the global GM map thus obtained is made up of:

- a list of characteristic points that represent valid references for the localization algorithm and

- a sparse and discrete representation of invalid volumes.

Each logistic operation invalidates a volume of the global GM map. At this point a reading can add features to the global GM map only if they fall within volumes registered as invalidated. This operation creates a new global reference in the global GM map and simultaneously validates the volume of the map. Conversely, if a reading indicates that an invalidated area is empty, this is also validated.

With this criterion, the observations from the cameras 2 and the notifications from the presence sensors 7 keep both the global map GM and the invalidation volumes map VI dynamically updated in such a way as to prevent spurious changes due to noise in the readings and guarantee its accuracy over time.

By using a dynamic GM global map, it is indeed vital for the localization system 1 not to introduce errors during the update process. Otherwise, after long periods of execution the errors will integrate and over time the GM global map will deviate from the truth.

In this way, therefore, the proposed localization system 1 provides at any time a reliable localization of the vehicles V, guaranteed by an updated global map GM of the references and assumed as true.

In fact, this prevents errors during the acquisition phases from leading to incorrect changes to the global GM map.

Furthermore, the accumulation of errors over time would impact the performance of the localization system 1, as the new incorrect version of the global GM map would still be used in subsequent matching searches.

Preferably, the localization system 1 comprises a plurality of presence sensors 7 installed on the gripping means F of the vehicle V, configured to detect the presence/absence of material M on the gripping means F and to detect at least an approximation of the volume occupied by the material M. In particular, the presence of additional presence sensors 7 on board the vehicle V allows the invalidation volume VI subject to changes to be detected with greater accuracy and sensitivity.

Therefore, this allows us to generate an invalidation volume VI to be added to the global GM map that has dimensions as close as possible to the volume actually occupied by the unloaded or loaded M material.

Furthermore, the use of different types of sensors in addition to presence sensors is not excluded.

For example, with reference to a vehicle consisting of a forklift, an additional height sensor 9 placed on the forks is able to measure at what height the material is moved.

According to a possible embodiment of the localization system 1, the local references LR and the global references GR comprise characteristic points (features) recognizable by the processing unit 4 from different angles.

It is in fact necessary that the same feature is observable and recognizable by the processing unit 4 from different perspectives.

Therefore, the above step of determining a plurality of local coordinates relative to a plurality of local references LR comprises processing the acquired image to locate feature points.

In particular, each feature point synthesizes the data provided by the acquisition means 2 for a particular object and provides a well-defined position in space.

Preferably, feature points include at least one of the following parameters: orientation, size, color, color gradient.

These features serve to facilitate the search for correspondences between the local references LR of the local map and the global references GR of the global map GM during the above-mentioned step of determining the position of the vehicle V.

In particular, the step of identifying global GR references on the global GM map corresponding to at least some of the determined local LR references comprises comparing and detecting correspondences between the feature points of the local LR references and the feature points of the global GR references present in the global GM map.

According to a possible and preferred embodiment, the step of identifying global references GR on the global map GM corresponding to at least some of the determined local references LR comprises at least the following steps:

- consider at least two local LR references and evaluate the distances and mutual orientations between the considered local LR references;

- group the GR global references into sets of the same number and type of the at least two LR local references considered and evaluate the distances and reciprocal orientations between the grouped GR global references of each set;

- compare the said at least two local references LR considered with all possible global references GR grouped and consider only those that have the same distances and reciprocal orientations (preferably in accordance with appropriate tolerances).

Therefore, local and global references are not compared to each other individually, but as pairs or more. This is because the position and orientation of a reference depend on the coordinates in which it is expressed, while the distance between two references or their mutual orientation is the same for all coordinate systems, in particular it is the same in both the local and global maps.

Therefore, the observed feature points are compared with known references present in the global GM map made available through the localization system 1.

The global GM map is itself made up of feature points that are directly comparable with observed ones.

Localization system 1 searches for the largest number of local features to match the global ones.

Furthermore, in order to determine the reliability of the solution, the localization system 1 provides a criterion based on the quality of the references that discards the solutions if they have a limited number of matches and/or are composed of references of low relevance. For each solution found, the processing unit 4 is configured to perform at least the following steps:

- for each observed local LR reference, associate a relevance value that represents both the quality of its detection and the sporadicity of its possible observation within area A;

- calculate the relevance of the solution as the sum of the relevances of each local reference LR whose correspondence with a global reference GR has been determined.

A vehicle V is defined as localized if it has a sufficient number of correspondences between local references LR and global references GR and if the solution has a sum of relevances greater than a certain threshold value.

The threshold value to discard solutions with too low relevance can be dynamic over time and depend on the vehicle situation.

In particular, a vehicle whose position is unknown starts with a much higher threshold level. As soon as a more or less precise hypothesis about the position and orientation of the vehicle is available, this threshold can be reduced. At the same time, solutions that are too far from the hypothesis are prohibited. In this way, the size of the global map allowed is reduced and consequently the possibility of encountering ambiguous solutions. Finally, the ability to estimate a hypothesis of the position before executing the algorithm can significantly improve computational performance. In fact, knowing the area of the plant where the vehicle may be present, it is possible to ignore references outside this area. In this way, the search for correspondences is significantly lightened.

Furthermore, the start of the localization system 1 requires the manual setting of the origin of the coordinates and the activation of a transitory and temporary self-learning phase, during which changes to the global map are permitted that are not justified by loading/unloading events. Finally, the localization system 1 can optionally include fixed fiducial references placed within the industrial area at the time of installation and kept unchanged over time.

For example, such markers can be printed on highly visible surfaces, bearing clearly distinguishable patterns that are unique for each reference. In such a hybrid configuration, the position estimation is performed by comparing both the material and the markers observed in the circumstances. The matches found must satisfy both reference maps, i.e. the static one composed of the markers and the dynamic one composed of the stored material.

In this way it is possible to locate the vehicle in those areas of the plant where the material is insufficient or completely absent.

In practice, it has been found that the described invention achieves the proposed purposes.

In particular, it is highlighted that the proposed solution introduces a system that is able to locate a vehicle through natural references present in the environment, without requiring either the installation of references at start-up or their subsequent maintenance.

In particular, the references detected and used are the material stored inside the warehouse.

The innovative aspect of the solution lies in particular in the dynamic management of the reference map.

In fact, in all known solutions there are two execution phases: the detection of local references and the localization through the search for correspondences between the detected references and the known ones in the global map. The localization process in fact involves determining the function that maps the local coordinates to the global ones. All known localization systems require the creation and management of a map of global references that remains constant over time. Changes in the references, such as addition, removal or movement, require human intervention. Otherwise, the system is not able to determine whether the deviations between what is known and what is observed are due to a reading error or an inaccuracy of the global map.

In a different way, the localization system according to the invention allows the localization of a vehicle through natural references present within the environment, without requiring either the installation of references at start-up or their subsequent maintenance.

Claims (12)

1) Localisation system (1) for industrial vehicles, comprising: - acquisition means (2) installed on at least one industrial vehicle (V) to be localised and configured to acquire at least one image (I) of the surrounding environment and/or data relating to the distances of objects surrounding said vehicle (V); - at least one storage unit (3) comprising at least one global map (GM) of an industrial area (A) within which said vehicle (V) must move; - at least one processing unit (4) operationally connected to said acquisition means (2) and to said storage unit (3) and configured to determine the position of said vehicle (V) within said area (A) starting from said at least one image (I) and/or from said distance data; characterized in that said global map (GM) comprises global coordinates relating to a plurality of global references (GR), wherein said global references (GR) correspond to fixed structural elements (E) present within said area (A) and to material (M) deposited and immobile within said area (A), and wherein said global coordinates are determined with respect to a predefined point of origin (O) on said global map (GM), and from the fact that said processing unit (4) is configured to iteratively repeat the following steps: - starting from said at least one acquired image (I) and/or from said distance data, determine a plurality of local coordinates relating to a plurality of local references (LR), where said local references (LR) correspond to fixed structural elements (E) present within said area (A) and/or to material (M) deposited and immobile within said area (A), and where said local coordinates are determined considering a point of origin on said vehicle (V); - identify global references (GR) on the global map (GM) corresponding to at least some of the said determined local references (LR); - starting from said identified global references (GR), determine the position (P) of said vehicle (V) within said area (A) by transforming the local coordinates of the vehicle (V) into corresponding global coordinates within the global map (GM); - once the said position (P) of the vehicle (V) has been determined, if at least one of the said local references (LR) determined does not correspond to any global reference (GR) on the global map (GM) and/or if on the global map (GM) there is at least one global reference (GR) which is not present among the local references (LR) determined, update the said global map (GM) by adding/deleting at least one global reference (GR). 2) Localization system (1) according to claim 1, characterized in that said processing unit (4) is configured to determine the orientation (D) of said vehicle (V) starting from said position (P) of the vehicle (V) and from the global coordinates of said identified global references (GR). 3) Localization system (1) according to one or more of the preceding claims, characterized in that it comprises at least one presence sensor (8) installed on the means (F) for gripping the material (M) of said vehicle (V) and operationally connected to said processing unit (4), said presence sensor (8) being configured to detect the presence/absence of material (M) on said gripping means (F). 4) Localization system (1) according to claim 3, characterized in that said processing unit (4) is configured to perform the following steps: - detect by means of said presence sensor (8) a loading/unloading operation of material (M) on/from said vehicle (V); - record the position (P) of said vehicle (V) during said unloading/loading operation; - starting from said recorded position (P) of the vehicle (V), determine an unloading/loading position of said material (M); - at said unloading/loading position, generate an invalidation volume (VI) within said global map (GM). 5) Localization system (1) according to claim 4, characterized in that said step of updating the global map (GM) comprises the following steps: - if said global reference (GR) to be added/removed from the global map (GM) corresponds to a generated invalidation volume (VI), add/delete said global reference (GR) to/from the global map (GM); - if said global reference (GR) to be added/removed from the global map (GM) does not match a generated invalidation volume (VI), do not add/delete said global reference (GR) to/from the global map (GM). 6) Localization system (1) according to one or more of claims 3 to 5, characterized in that it comprises a plurality of presence sensors (8) installed on said gripping means (F) of the vehicle (V), said presence sensors (8) being configured to detect the presence/absence of material (M) on said gripping means (F) and to detect at least an approximation of the volume occupied by said material (M). 7) Localization system (1) according to one or more of the preceding claims, characterized in that said step of identifying global references (GR) on the global map (GM) corresponding to at least some of the determined local references (LR) comprises at least the following steps: - consider at least two local references (LR) and evaluate the distances and mutual orientations between the local references (LR) considered; - group said global references (GR) into sets of the same number and type as said at least two local references (LR) considered and evaluate the distances and reciprocal orientations between said grouped global references (GR) of each set; - compare the said at least two local references (LR) considered with all the possible global references (GR) grouped and consider only the global references (GR) that have the same distances and mutual orientations as the said at least two local references (LR). 8) Localization system (1) according to one or more of the preceding claims, characterized in that said local references (LR) and said global references (GR) comprise characteristic points recognizable from different angles. 9) Localization system (1) according to claim 8, characterized in that said step of determining a plurality of local coordinates relative to a plurality of local references (LR) comprises processing said at least one acquired image (I) to identify said characteristic points. 10) Localization system (1) according to one or more of claims 8 and 9, characterized in that such characteristic points comprise at least one of the following parameters: orientation, size, color, color gradient. 11) Localization system (1) according to one or more of claims 8 to 10, characterized in that said step of identifying global references (GR) on the global map (GM) corresponding to at least some of said determined local references (LR) comprises comparing and detecting correspondences between the characteristic points of the local references (LR) and the characteristic points of the global references (GR) present in the global map (GM). 12) Localization system (1) according to one or more of the preceding claims, characterized in that said acquisition means (2) comprise at least one of a stereoscopic camera or a LIDAR system.