RU2658092C2 - Method and navigation system of the mobile object using three-dimensional sensors - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: this technical solution refers, generally, to power engineering, to computing systems and methods, and in particular to the systems and methods for navigating the mobile objects using three-dimensional sensors. Method for navigating the mobile object using three-dimensional sensors is characterized by the fact that the data of the three-dimensional scene of the storage room are obtained from at least one three-dimensional sensor, which is installed on the mobile object, and the known true coordinates of the rack points are obtained in advance; the set of hypothetical coordinates of the rack points based on the data of the three-dimensional scene of the warehouse is formed; the hypothetical coordinates of the location of the mobile object are determined; fix precise determination of the coordinates of the location of the mobile object is performed based on the comparison of the hypothetical coordinates of the rack points and the predetermined coordinates of the rack points that are obtained above.

EFFECT: technical result of inventions consists in increasing the accuracy of navigation of the mobile object inside the storage room.

11 cl, 6 dwg

Description

FIELD OF TECHNOLOGY

[0001] This technical solution relates, in general, to computer systems and methods, and in particular to systems and methods for navigating moving objects using three-dimensional sensors.

BACKGROUND

[0002] US 20130096735 A1, "Warehouse vehicle navigation system and method", patent holder: Vocollect, Inc., publication date: 04/18/2013 is known from the prior art.

[0003] The navigation system described in the patent is based on a video camera, and a storage vehicle in this case provides for the mandatory presence of a person on board. Also, although the name declares the concept of a warehouse robot in the broad sense, the fact is that the description relates to a transport trolley for the “picking” procedure (collecting a small wholesale or retail order from the first tier of racks). The method and navigation system require physically marking each rack with a unique digital code (i.e., in fact, with a unique marker), and through the video camera this code is recognized and the location of the robot is identified. Thus, navigation is carried out in longitudinal motion along the racks. The distance to the rack, according to the method, is determined by detecting the affinity conversion of the rectangular marker card.

[0004] Due to the lack of image correction in this technical solution, the error of the data increases. The disadvantages include the fact that the technical solution assumes a near-ideal frontal location of the marker in front of the camera, which is difficult to achieve in the real conditions of a warehouse.

SUMMARY OF THE INVENTION

[0005] This technical solution is aimed at eliminating the disadvantages inherent in solutions known from the prior art.

[0006] The technical problem to be solved in this technical solution is to create a method for navigating a moving object inside a warehouse, fixing the a priori known structure of the rack.

[0007] The technical result is to increase the accuracy of navigation of a moving object inside a warehouse. Achievement of the indicated technical result was proved in the framework of testing this technical solution on the data taken at a really functioning distribution warehouse, in particular, the proposed technical solution allowed to improve navigation accuracy by 7 cm (error reduced from 10 cm to 3 cm). This is illustrated in FIG. 6.

[0008] Additionally, the reliability of the system.

[0009] The indicated technical result is achieved by a method for navigating a moving object using three-dimensional sensors, in which three-dimensional scene data of a warehouse from at least one three-dimensional sensor mounted on a moving object and the previously known true coordinates of the shelving points are obtained; form a set of hypothetical coordinates of the points of the shelves based on the data of the three-dimensional scene of the warehouse; determine the hypothetical coordinates of the location of the moving object; performing a refinement of the coordinates of the location of the moving object based on a comparison of the hypothetical coordinates of the points of the shelves and the previously known true coordinates of the points of the shelves obtained above.

[00010] In some embodiments of the present technical solution, the three-dimensional sensor is a time-of-flight sensor or a triangulation scanner or a laser scanning system.

[00011] In some embodiments of the present technical solution, the three-dimensional scene data is a point cloud of the scene.

[00012] In some embodiments of the present technical solution, the previously known true coordinates of the shelving points on a wireless or wired data channel are obtained.

[00013] In some embodiments of the present technical solution, the previously known true coordinates of the shelving points obtained by the operator via the data input / output device are obtained.

[00014] In some embodiments of the present technical solution, the previously known true coordinates of the points of the shelves are three-dimensional.

[00015] In some embodiments of the present technical solution, when forming a set of shelving fronts on a horizontal plane, a point cloud is converted to a raster image.

[00016] In some embodiments of the present technical solution, the hypothetical location coordinates of the moving object are the location coordinates of the moving object at the previous location where the moving object was located.

[00017] In some embodiments of the present technical solution, the hypothetical location coordinates of the moving object are obtained from another navigation system.

[00018] In some embodiments of the present technical solution, the location coordinates of the moving object are updated iteratively when the moving object is moving.

[00019] Also, the above technical result is achieved due to the navigation system of a moving object using three-dimensional sensors, comprising: at least one three-dimensional sensor mounted on a moving object and configured to receive three-dimensional scene data of a warehouse; at least one processing component; a memory for storing instructions executed by the processing component, the processing component being configured to receive three-dimensional scene data of the warehouse from the three-dimensional sensor and the previously known true coordinates of the shelving points; the formation of a set of hypothetical coordinates of the points of the shelves based on data from a three-dimensional scene of the warehouse determining hypothetical coordinates of the location of the moving object; performing refinement of the coordinates of the location of the moving object based on a comparison of the hypothetical coordinates of the points of the racks and the previously known true coordinates of the points of the racks obtained above.

BRIEF DESCRIPTION OF THE DRAWINGS

[00020] The features and advantages of this technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

[00021] In FIG. 1 shows a flowchart of a method for navigating a moving object using three-dimensional sensors;

[00022] FIG. 2 shows an example of the approach of a moving object to the front of the rack;

[00023] In FIG. 3 shows an example of projecting a three-dimensional cloud of points onto a two-dimensional horizontal plane;

[00024] FIG. Figure 4 shows an example of a raster image obtained by projecting a cloud of points on a horizontal plane (upper part of the figure), as well as a raster image of the front of the rack (lower part of the figure);

[00025] In FIG. 5 shows an example implementation of a navigation system for a moving object using three-dimensional sensors.

[00026] In FIG. Figure 6 shows a bar graph with the accuracy characteristics of a technical solution, tested on three-dimensional data taken in a really functioning warehouse, with artificial noise of the reference / true coordinate. Two configurations of the technical solution were tested, and each configuration has four values of artificial noise of the specified coordinate (a total of 8 values). The diagram shows that with a maximum error of the coordinate being refined of 10 cm, the technical solution improves it to 3 cm. If the error is minimal and 3 cm, the technical solution improves it slightly - up to 2.5 cm. It should be noted that the obtained navigation accuracy is minimal in the prior art and allows the control system of a moving object to qualitatively work out the required tasks.

DETAILED DISCLOSURE OF TECHNICAL SOLUTION

[00027] Below will be described the concepts and definitions necessary for the detailed disclosure of the ongoing technical solution.

[00028] The technical solution may be implemented as a distributed computer system.

[00029] In this solution, a system means a computer system, a computer (electronic computer), CNC (numerical program control), PLC (programmable logic controller), computerized control systems and any other devices that can perform a given, well-defined sequence of operations (actions, instructions).

[00030] An instruction processing device is understood to mean an electronic unit or an integrated circuit (microprocessor) executing machine instructions (programs).

[00031] An instruction processing device reads and executes machine instructions (programs) from one or more data storage devices. Storage devices may include, but are not limited to, hard disks (HDD), flash memory, ROM (read only memory), solid state drives (SSD), optical media (CD, DVD, etc.).

[00032] A program is a sequence of instructions for execution by a computer control device or an instruction processing device.

[00033] A moving object is an autonomously moving technical object that operates in a warehouse. Such moving objects may include transport trolleys, pallet transporters, loading equipment of various kinematic schemes, equipped with various power units with different levels of human participation in the control process, unmanned aerial vehicles, not limited to.

[00034] Scene (Eng. Scene) - an image of three-dimensional space with objects located in it.

[00035] The image plane (eng. Image plan) is the plane on which a two-dimensional image of the rendered or rasterized scene is formed.

[00036] Rack - equipment for storing items, consisting of multi-tiered decks (shelves) mounted on racks or side walls, or consisting of consoles fixed on racks (cantilever racks).

[00037] Pallet racks (front) - front racks that are designed to store various goods on pallets.

[00038] A point cloud is a set of vertices in a three-dimensional coordinate system. These vertices, as a rule, are determined by the coordinates X, Y and Z and, as a rule, are intended to represent the outer surface of the object.

[00039] FIG. 1 is a flowchart showing a method for navigating a moving object using three-dimensional sensors, which comprises the following steps.

[00040] Step 101: obtaining three-dimensional scene data of the warehouse from at least one three-dimensional sensor mounted on the movable object, and pre-known true coordinates of the points of the shelves.

[00041] As a three-dimensional sensor, time-of-flight sensors, triangulation scanners, laser scanning systems, such as lidars, etc., can be used, without limitation. The data of a three-dimensional scene refers to the resulting point cloud of the scene. Additionally, you can receive images. The well-known true coordinates of the shelving points are obtained via a wireless or wired data channel, and they can also be entered by the operator via input / output devices. The coordinates of the shelving points are three-dimensional.

[00042] In some embodiments, a set of shelving fronts are formed on a horizontal plane based on three-dimensional scene data obtained in the previous step.

[00043] At this step, the previously obtained point cloud is projected onto a horizontal plane and converted to a raster image (Fig. 3). The advantages of a raster image format are obvious - they are easy to edit (crop, resize, resize an image, etc., etc.). Projection on a horizontal plane means the exclusion of the vertical coordinate (Z) of a point cloud. Such a projection can be interpreted as a projection onto a horizontal floor or horizontal ceiling for a clearer understanding. Next, the resulting raster image is processed by a filter for smoothing pixel intensities. In some embodiments, a Gaussian filter, a filter over a sliding normalized window, a median filter, a bilateral filter can be used. Then, the boundaries are detected on the image, which can be performed using the Canny algorithm, the Sobel algorithm, the Prewitt operator, the Roberts operator, not limited to, then discard the lines whose length is less than a predetermined threshold, and compare the set of received lines with the predetermined geometric parameters of the shelving and the shelving space . In some embodiments, the width of the corridor is known along which the desired shelving stands to the right and left. The distance in meters between the parallel lines defined on the raster image is measured (by multiplying the number of pixels by the coefficient obtained during the formation of the raster image, i.e. how many meters / millimeters per pixel). This distance with a small error corresponds to the known width of the above corridor. In some embodiments, this may be the distance between other structural elements of the rack, distinguishable in a plan view. Thus, each found line in the horizontal projection (Fig. 4) represents the front of the rack (front of the rack), and the set of lines received is a set of fronts of the shelves.

[00044] Step 102: form a set of hypothetical coordinates of the points of the racks based on the data of the three-dimensional scene of the warehouse;

[00045] In the next step, a return is made from a two-dimensional horizontal projection along the selected front of the rack to a three-dimensional point cloud. To do this, a set of selected lines in two-dimensional form after filtering is transferred to a three-dimensional point cloud. Only in this case, not the entire cloud of points is used, but only those points that belong to the found front of the rack. After that, a cloud of points of the front of the rack is projected into the frontal vertical plane (X and Z coordinates are taken into account) and the resulting projection is converted to a raster image. An example of such a bitmap is shown at the bottom of FIG. 4. Then create an image template of the racks (the vertical element of the rack) and the beams (the horizontal element of the rack) based on the predetermined geometric parameters of the typical constructive structures of the racks. The template is a reference two-dimensional raster image of the ideal section of the structural element in the frontal projection, intended for the further correlation procedure. Then, each template is correlated by a raster image of the vertical front of each rack and correlation maxima are found.

[00046] In some embodiments of the technical solution, correlation is performed, for example, by the square of the difference:

[00047] where R (x, y) is the intensity of the pixel of the correlation matrix; T (x ', y') - template pixel intensity; I (x + x ', y + y') is the pixel intensity of the bitmap image or, in the general case, the image on which the feature described by the template is searched for.

[00048] The correlation maxima may be represented by a correlation matrix. The maxima of the obtained correlation matrix and will correspond to the hypothetical coordinates of the rack. Next, the coordinates of the correlation maxima of the raster image are converted into three-dimensional coordinates, thereby obtaining a set of hypothetical coordinates of the rack racks.

[00049] Step 103: determine the hypothetical location coordinates of the moving object.

[00050] In some embodiments, the hypothetical coordinates, which are further specified, are the coordinates of the location of the moving object at the previous location and are known in advance in the system.

[00051] In some embodiments, hypothetical coordinates of the location of a moving object are determined by obtaining them from another navigation system operating with large errors, for example, by using odometry or an inertial navigation system.

[00052] Step 104: refine the location coordinates of the moving object based on a comparison of the hypothetical coordinates of the shelving points and the predetermined known true coordinates of the shelving points obtained above.

[00053] In this step, the hypothetical coordinates of the shelving points are first compared with the previously known true coordinates of the shelving points using the closest neighbor method. For example, hypothetical points G1, G2, and G3 are found around a previously known true rack point T. The closest point is determined by using, for example, the Euclidean metric, for example, G3. Thus, in this embodiment, for a previously known true point T, point G3 is mapped using the closest neighbor method.

[00054] The above technique is carried out for all previously known true shelf points. That is, they take the previously known true coordinate of the rack point in a flat horizontal space and find, by means of the Euclidean metric, not limited to, the closest hypothetical coordinate.

и координат картированных стоек

, а также квадрат разности искомой навигационной координаты подвижного объекта по найденным стойкам р и координаты предварительной р0.

,

, K₀ - ковариации соответственно искомого положения, картирования и предыдущего местоположения подвижного объекта.[00055] Then form the functional of the sum of the variances of the hypothetical coordinates of the points of the shelves from the previously known true coordinates of the points of the shelves and at least one coordinate of the moving object. A functional is a formula that determines the quality of the description of the found hypothetical coordinates of the rack, previously known (mapped). It includes the square of the difference of the hypothetical coordinates of the racks

and coordinates of mapped racks

, as well as the square of the difference of the desired navigation coordinate of the moving object by the found struts p and the coordinates of the preliminary p0.

,

, K ₀ - covariance, respectively, of the desired position, mapping and previous location of the moving object.

[00056] Procedure for updating the location coordinates of a moving object

performed iteratively with a high frequency during the movement of a moving object.

[00057] Then, an updated coordinate of the position of the moving object from the first derivative of the functional is determined by the coordinates of the moving object, for example, by the following formula

[00058] FIG. 5 is a block diagram showing a navigation system of a moving object using three-dimensional sensors, which comprises:

[00059] at least one three-dimensional sensor mounted on a moving object and configured to receive three-dimensional scene data of a warehouse;

[00060] at least one processing component;

[00061] a memory for storing instructions executed by the processing component,

[00062] wherein the processing component is configured to:

[00063] obtaining three-dimensional scene data of the warehouse from the three-dimensional sensor and the previously known true coordinates of the points of the racks;

[00064] generating a set of shelving fronts on a horizontal plane based on three-dimensional scene data obtained in the previous step;

[00065] the formation of a set of hypothetical coordinates of the points of the racks based on the data of a three-dimensional scene of the warehouse and a set of fronts of the racks;

[00066] determining hypothetical location coordinates of a moving object;

[00067] performing the refinement of the coordinates of the location of the moving object based on a comparison of the hypothetical coordinates of the points of the racks and the predetermined coordinates of the points of the racks obtained above.

[00068] In some embodiments, system 500 may be a mobile phone, a computer, a messaging device, a tablet, and a personal digital assistant, etc.

[00069] Referring to Figure 5, system 500 may include one or more of the following components: processing component 502, three-dimensional sensor 503, memory 504, power component 506, multimedia component 508, input / output (I / O) interface 512, touch component 514, data transmission component 516.

[00070] In some embodiments, the processing component 502 mainly controls all operations of the system 500, for example, display, data transfer, camera operation, and recording operation. The processing component 502 may include one or more processors 518 that implement instructions to complete all or part of the steps of the above methods. In addition, the processing component 902 may include one or more modules for a convenient interaction process between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module for conveniently facilitating interaction between the multimedia component 508 and the processing component 502.

[00071] As a three-dimensional sensor 503, any technology known from the prior art for obtaining and processing information about remote objects using active optical systems using the phenomena of light reflection and scattering in transparent and translucent media can be used. For example, it can be a time-of-flight sensor, a triangulation scanner, a laser scanning system, for example, lidar.

[00072] The memory 504 is configured to store various types of data to support the operation of the system 500. Examples of such data include instructions from any application or method, image, video, etc. Memory 504 may be implemented as any type of volatile memory, non-volatile memory, or a combination thereof, for example, Static Random Access Memory (RAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Permanent Storage Device (ROM), Permanent Storage Device (ROM), magnetic memory, flash memory, magnetic or an optical disc.

[00073] In some embodiments, power component 506 provides electricity to various components of system 500. Power component 506 may include a power management system, one or more power supplies, and other nodes for generating, controlling, and distributing power to system 500.

[00074] In some embodiments, the multimedia component 508 includes a screen providing an output interface between the system 500 and the user. In some embodiments, the implementation of the screen may be a liquid crystal display (LCD) or touch panel (SP). If the screen includes a touch panel, the screen may be implemented as a touch screen for receiving an input signal from a user. The touch panel includes one or more touch sensors in the sense of gesturing, touching and sliding the touch panel. The touch sensor can not only feel the touch or the gesture of turning over, but also determine the duration of time and pressure associated with the operation mode of touch and sliding.

[00075] An I / O interface 512 provides an interface between the processing component 502 and the peripheral interface module.

[00076] The sensor component 514 contains one or more sensors and is configured to provide various aspects of assessing the state of the system 500. For example, the sensor component 514 can detect on / off states of the system 500, the relative position of components, for example, the display and keypad of the device 500, changing the position of the system 500 or one component of the system 500, the presence or absence of contact between the user and the system 500, as well as the orientation or acceleration / deceleration and changing the temperature of the system 500. Touch component Entent 514 contains a proximity sensor configured to detect the presence of an object in the vicinity when there is no physical contact. The sensor component 514 includes an optical sensor (e.g., CMOS or CCD image sensor) configured to be used in visualizing an application. In some embodiments, the sensor component 514 comprises an acceleration sensor, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[00077] The communication component 516 is configured to facilitate wired or wireless communication between the system 500 and other devices. System 500 may access a wireless network based on a communication standard such as WiFi, 2G, or 3G, or a combination thereof. In one exemplary embodiment, the data transmission component 516 receives a broadcast signal or broadcast, associated information from an external broadcast control system via a broadcast channel. In one embodiment, the data transmission component 516 comprises a near field communication (NFC) module to facilitate near field communication. For example, the NFC module can be based on radio frequency identification (RFID) technology, infrared data association technology (IrDA), ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[00078] In an exemplary embodiment, the memory 504 includes instructions that are executed by a processor 518 of the system 500 to implement the above methods of measuring distance to remote objects. For example, a non-volatile computer-readable medium may be a ROM, random access memory (RAM), compact disk, magnetic tape, floppy disks, optical storage devices and the like.

[00079] A person skilled in the art can readily understand other embodiments of the invention from the foregoing description. This application is intended to cover any use or application of the following general principles of the invention, and including those deviations from the present invention that appear within the scope of known or ordinary practice in the art. It is assumed that the description is considered only as an example, with the essence and scope of the present invention indicated by the claims.

[00080] It should be appreciated that the present invention is not limited to the precise structures that have been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made without departing from its scope. It is intended that the scope of the invention be limited only by the appended claims.

[00081]

Claims (23)

1. The method of navigation of a moving object using three-dimensional sensors, characterized in that: - receive data from a three-dimensional scene of a warehouse from at least one three-dimensional sensor mounted on a moving object, and the previously known true coordinates of the points of the shelves; - form a set of hypothetical coordinates of the points of the shelves based on the data of the three-dimensional scene of the warehouse; - determine the hypothetical coordinates of the location of the moving object; - perform the refinement of the coordinates of the location of the moving object based on a comparison of the hypothetical coordinates of the points of the racks and the predetermined coordinates of the points of the racks obtained above. 2. The method according to p. 1, characterized in that the three-dimensional sensor is a time-of-flight sensor, or a triangulation scanner, or a laser scanning system, or a stereo camera, or a depth sensor. 3. The method according to p. 1, characterized in that the data of the three-dimensional scene is a point cloud of the scene. 4. The method according to p. 1, characterized in that receive in advance the known true coordinates of the points of the shelves via a wireless or wired data channel. 5. The method according to p. 1, characterized in that they receive a pre-known true coordinates of the points of the shelves, entered by the operator through the device input / output data. 6. The method according to p. 1, characterized in that the previously known true coordinates of the points of the shelves are three-dimensional. 7. The method according to p. 1, characterized in that when forming a set of shelving fronts on a horizontal plane, a point cloud is converted to a raster image. 8. The method according to p. 1, characterized in that the hypothetical coordinates of the location of the moving object are the coordinates of the location of the moving object at the previous location in which the moving object was located. 9. The method according to claim 1, characterized in that the hypothetical coordinates of the location of the moving object are obtained from another navigation system. 10. The method according to p. 1, characterized in that they refine the coordinates of the location of the moving object iteratively when moving the moving object. 11. The navigation system of a moving object using three-dimensional sensors, contains: - at least one three-dimensional sensor mounted on a moving object and configured to receive data of a three-dimensional scene of a warehouse; at least one processing component; - a memory for storing instructions executed by the processing component, the processing component being configured to: - receiving data of a three-dimensional scene of a warehouse from a three-dimensional sensor and previously known true coordinates of the points of the shelves; - forming a set of shelving fronts on a horizontal plane based on the data of a three-dimensional scene obtained in the previous step; - the formation of a set of hypothetical coordinates of the points of the racks based on the data of the three-dimensional scene of the warehouse and a set of fronts of the racks; - determination of hypothetical coordinates of the location of the moving object; - performing refinement of the coordinates of the location of the moving object based on a comparison of the hypothetical coordinates of the points of the shelves and the predetermined coordinates of the points of the shelves obtained above.

Images