TW202124990A - State estimation and sensor fusion methods for autonomous vehicles - Google Patents

Abstract

State estimation and sensor fusion methods for autonomous vehicles thereof are provided. The autonomous vehicle includes at least one sensor, at least one motor and a processor, and is configured to transfer and transport an object. In the method, a task instruction for moving an object and data required for executing the task instruction are received. The task instruction is divided into a plurality of work stages according to respective mapping locations, and each of the work stages is mapped to one of a transport state and an execution state, so as to establish a semantic hierarchy. A current location of the autonomous vehicle is detected by using the sensor and mapped to one of the work stages in the semantic hierarchy, so as to estimate a current state of the autonomous vehicle.

Description

Mobile vehicle and its state estimation and sensing fusion switching method

The present invention relates to a method for estimating the state of a device, and more particularly to a mobile vehicle and a method for fusion switching of state estimation and sensing.

Automated Guided Vehicle (AGV) is a mobile robot that can move goods in factories and warehouses through technologies such as floor wires, machine vision, or laser navigation. Because AGV can automatically load, unload and transport goods, it saves effort on loading and unloading, and can flexibly allocate loading and unloading locations and transportation routes to improve delivery efficiency and solve problems such as lane occupation.

AGV relies on technologies such as positioning and object recognition to carry out cargo handling. In recent years, a variety of positioning technologies have come together, such as Bluetooth, WiFi, Ultra-Wideband (UWB), and visible light positioning system ( Visible Light Positioning System), Radio Frequency Identification (RFID), etc., depending on deployment cost, accuracy, and technical characteristics, each of these positioning technologies has its own field of application. Due to the diversity of positioning technology, the design of seamless indoor and outdoor positioning is difficult to achieve by simply switching between dual systems.

The purpose of the present invention is to provide a mobile vehicle and its state estimation and sensing fusion switching method, which can realize seamless switching between multiple positioning systems.

The invention provides a method for switching between state estimation and sensing fusion of a mobile carrier. The mobile carrier includes at least one sensor, at least one actuator and a processor for transferring and transporting objects. This method includes the following steps: receiving a task instruction for moving objects and the data required to execute the task instruction; dividing the task instruction into multiple work stages according to the mapping position, and mapping each work stage to one of the transportation state and the execution state 1. To establish a semantic level; use the sensor to estimate the current position of the mobile vehicle; and map this current position to one of the working stages in the semantic level to estimate the current state of the mobile vehicle.

The invention provides a mobile vehicle, which includes a data capture device, at least one sensor, at least one actuator, a storage device, and a processor. Among them, the sensor is used to estimate the current position of the mobile vehicle. The actuator is used to transfer and transport objects. The storage device is used to store the data captured by the data capture device and multiple computer commands or programs. The processor is coupled to the data capture device, the sensor, the actuator, and the storage device, and is configured to execute computer commands or programs to: use the data capture device to receive task instructions for moving objects and execute the task instructions Data; divide the task command into multiple work phases based on the mapping position, and map each work phase to one of the transportation state and the execution state to establish a semantic level; and map the current position estimated by the sensor to the semantics One of the working stages in the hierarchy to estimate the current state of the mobile vehicle.

The mobile vehicle and its state estimation and sensing fusion switching method of the present invention establish a semantic level by dividing task commands into multiple work phases and mapping them to different states. When the mobile vehicle is performing transfer and transportation of objects During the task, it is possible to map the estimated position to the current state and determine whether a state transition occurs. When a state transition occurs, it can also quickly switch to a sensing combination suitable for the current state to continue executing task commands. Thereby, the state estimation of the mobile vehicle and its sensing fusion switching can be performed efficiently, and seamless switching between positioning systems can be realized.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail in conjunction with the accompanying drawings.

The embodiment of the present invention designs a common architecture for autonomous mobile vehicles (Automated Guided Vehicle, AGV), in which the received task instructions are divided into multiple work stages according to their mapping positions to establish a semantic hierarchy, and then Each work stage is mapped to the state layer along with the semantic level according to its sequence and connection relationship to establish a state transition model. In real-time operation, autonomous mobile vehicles can estimate their current position and map the position to the semantic level to estimate the current state. In addition, the autonomous mobile vehicle can compare the difference between the current state and the previous state to determine whether a state transition occurs, and re-prioritize the sensors when a state transition occurs, so as to efficiently switch to the current state Control thread to continue the handling work.

FIG. 1 is a block diagram of a mobile vehicle according to an embodiment of the present invention. Please refer to FIG. 1, the mobile carrier 10 of this embodiment is, for example, an electronic device such as an autonomous mobile carrier, a handling robot, etc., used to transfer and transport objects. The mobile vehicle 10 includes a data acquisition device 12, at least one sensor 14, at least one actuator 16, a storage device 18, and a processor 20, and its functions are described below.

The data capture device 12 is, for example, an interface device such as a universal serial bus (USB) interface, a Firewire interface, a Thunderbolt interface, a card reader, etc., which can be used to connect external devices such as flash drives, mobile hard drives, memory cards, etc. Device to retrieve data. In another embodiment, the data capture device 12 is an input tool such as a keyboard, a mouse, a touchpad, a touch screen, etc., for detecting the input operation of the user to capture input data. In another embodiment, the data acquisition device 12 is, for example, a network card that supports wired network connections such as Ethernet or supports Institute of Electrical and Electronics Engineers (IEEE) 802.11 A wireless network card with wireless communication standards such as n/b/g, which can connect to an external device through a wired or wireless network and retrieve data.

The sensor 14 is, for example, a wireless communication subsystem, a global positioning system (GPS), a Bluetooth low energy (Bluetooth Low Energy, BLE), an inertial measurement unit (IMU), a rotary encoder (rotary encoder) encoder), camera, photodetector (photodetector), laser, or a combination thereof, and can sense environmental information such as electromagnetic waves, images, and sound waves around the mobile vehicle 10, as well as the inertia and displacement of the mobile vehicle 10 itself, and The detected information is provided to the processor 20 to estimate the current position and/or state of the mobile vehicle 10. In one embodiment, the sensor 14 can be used with systems such as laser mapper and odometry to increase the precise estimation of the position of the mobile vehicle 10.

The actuator 16 is, for example, a fork, an arm, a roller, a motor, or a combination thereof, which can form a fork-arm conveying system, and can be based on a control command issued by the processor 20 or Signals, loading, unloading and transporting objects.

The storage device 18 can be any type of fixed or removable random access memory (random access memory, RAM), read-only memory (read-only memory, ROM), flash memory (flash memory) or Similar elements or combinations of the above elements. In this embodiment, the storage device 18 is used to store the data retrieved by the data retrieval device 12 and computer commands or programs that can be accessed and executed by the processor 20. Among them, the data captured by the data capture device 12 includes task instructions, map data, identification information and other data required to execute the task instructions, and the processor 20 can use the map data to estimate the position, and use the identification information to transfer data. Perform identification operations on objects, loading or unloading locations, and loading or unloading objects. The method for identifying the loading object and the unloading object includes biological characteristics, object characteristics, environmental characteristics or identification codes, which are not limited here.

The processor 20 is, for example, a central processing unit (CPU) or a graphics processing unit (GPU), or other programmable general-purpose or special-purpose microprocessors (Microprocessor), digital signal processing Digital Signal Processor (DSP), programmable controller, Application Specific Integrated Circuits (ASIC), Programmable Logic Device (PLD) or other similar devices or a combination of these devices . The processor 20 is connected to the data acquisition device 12, the sensor 14, the actuator 16 and the storage device 18. For example, it loads computer instructions or programs from the storage device 18 and executes the state estimation of the mobile vehicle according to the present invention. Fusion switching method of sensing and sensing. The following examples illustrate the detailed steps of this method.

FIG. 2 is a flowchart of a method for fusion switching between state estimation and sensing of a mobile vehicle according to an embodiment of the present case. Please refer to FIGS. 1 and 2 at the same time. The method of this embodiment is applicable to the mobile carrier 10 of FIG. detailed steps.

In step S202, the processor 20 utilizes the data capture device 12 to receive the task instruction of the object to be transported and the data required to execute the task instruction. Wherein, the task instruction is, for example, issued by the manager of the factory area to instruct the mobile vehicle 10 to perform operations such as moving and transporting objects in the factory area. In one embodiment, the processor 20, for example, stores data that is frequently read or will be used in the storage device 18, such as maps and objects to be transported in nearby areas, loading or unloading locations, and identification information of loading or unloading objects in the storage device 18. In order to provide the processor 20 for access and use.

In step S204, the processor 20 divides the task instruction into multiple work phases according to the mapping position, and maps each work phase to one of the transportation state and the execution state to establish a semantic hierarchy. Wherein, the task instruction is composed of at least one task of loading, unloading, and transportation, and the processor 20, for example, corresponds these tasks to at least one control thread, and distinguishes the working stages according to the control thread. Wherein, the loading and unloading are based on, for example, the loading location, the unloading location, the transferred object, and the identification of the loading object and the unloading object to distinguish the working stages, and the transportation work is based on, for example, the respective locations of at least one location through which the transportation passes. Geographic Information System to distinguish work phases.

In one embodiment, the processor 20 classifies the state of the mobile vehicle 10 into two categories: transportation state or execution state. In the transportation state, the processor 20, for example, uses a path planner to set a path. The path planning module plans a path to construct a visibility map based on the method proposed by Ghosh and Mount, and based on the edges of the visibility map. , Using the shortest path algorithm such as Dijkstra's algorithm to calculate the best path, and generate low-level instructions to control the motor of the mobile vehicle 10 to adjust the direction and speed to track the planned path. During transportation, the processor 20 will use the sensor 14 to continuously sense the surrounding environment and confirm whether the mobile vehicle 10 is moving. When an obstacle is detected, the processor 20 controls the motor to slow down according to the distance measurement data. Or stop, and use the laser mapping system to survey and map the shape of the obstacle and output it to the path planning module to plan the obstacle avoidance path. On the other hand, in the execution state, the processor 20, for example, activates the camera to perform the recognition of the loading/unloading object, and controls the transfer machine to perform the loading and unloading of the article.

In detail, the state estimation and sensing fusion switching method of the mobile vehicle of this embodiment establishes a semantic level to give the cognitive system the ability when implementing state analysis. Among them, the semantic level can be dynamically established based on task instructions, including three levels of mapping position, work stage, and status.

For example, FIG. 3 is a schematic diagram illustrating the semantic level according to an embodiment of the present case. Please refer to FIG. 3, the semantic level 30 includes a mapping location layer 32, a work stage layer 34 and a status layer 36. The mapping location layer 32 includes areas or locations involved in executing task instructions, such as coordinates 1-3, map tiles 1-3, and (transfer location/object) images 1-3. The work phase layer 34 includes multiple work phases, for example, including loading P1, transporting P2 to P3, and unloading P4. Each position in the mapping position layer 32 can be mapped to one of loading P1, transportation P2~P3, and unloading P4. For example, coordinate 3 and image 3 can be mapped to loading P1, and coordinate 2, image 2 and image 3 can be mapped to unloading P4, and so on. The state layer 36 includes an execution state and a transportation state. The loading P1 and the unloading P4 can be mapped to the execution state, and the transportation P2 to P3 can be mapped to the transportation state. Various execution states and transportation states can correspond to a thread of a feedback control loop. This thread, for example, couples a specific sensor 14 and an actuator 16 to control it to perform a specific operation.

In one embodiment, after the processor 20 establishes the semantic level, for example, further according to the sequence and connection relationship between the work stages, each work stage is mapped to one of the transportation state and the execution state along with the semantic level to form State transition model (state transition model).

For example, FIG. 4 is a schematic diagram of a state transition model according to an embodiment of the present application. Referring to FIG. 4, the state transition model 40 defines, for example, the transition between various work stages in the transportation state and the execution state in the semantic hierarchy. That is, the state transition model 40 maps the transition between work phases to the transition between states. Taking Fig. 4 as an example, the state transition model 40 records the transitions between the work phases 1~n mapped to the transportation state, the transitions between the work phases 1~m mapped to the execution state, and the work phases 1~n and the work phases. Conversion between 1~m. The table on the bottom left maps to the sensors and actuators coupled to the working stage 1~n of the transportation state, and the table on the bottom right maps to the sensors and actuators coupled to the working stage 1~m of the execution state Device. For example, the working phase 1 mapped to the transportation state couples the global positioning system and the base station, the working phase 2 mapped to the transportation state couples the optical sensor, the inertial measurement unit, and the rotary encoder, and so on.

After the semantic level and the state transition model are established, in real-time operation, the mobile vehicle 10 can estimate its current position and map the position to the semantic level to estimate the current state.

In detail, in step S206, the processor 20 uses the sensor 14 to estimate the current position of the mobile vehicle 10. Among them, the processor 20 may use a global positioning system or a base station positioning system to estimate an outdoor position, or use a positioning device such as a light sensor or a laser to estimate an indoor position, and there is no limitation here.

Finally, in step S208, the processor 20 maps the current position to one of the working stages in the semantic hierarchy to estimate the current state of the mobile vehicle 10. Taking Fig. 3 as an example, when the processor 20 estimates the current position of the mobile vehicle 10 to obtain the coordinate 3, it can map the coordinate 3 to the load P1 in the working stage through the semantic level 30, and then map the load P1 to Execution status. Accordingly, the processor 20 can couple corresponding sensors and actuators to perform primary actions or skills according to its estimated current state.

After the processor 20 estimates the current state of the mobile vehicle 10, for example, it compares the current state with the previous state estimated at the previous time point to determine whether a state transition occurs. Wherein, when determining that a state transition occurs, the processor 20 sequentially switches a plurality of sensing combinations under the state transition according to the previously established state transition model, so as to select available sensing combinations to continue to execute the task instruction. The sensing combination includes at least one sensor and/or actuator. By rescheduling the combination of sensing signal sources during state transition, it is possible to efficiently switch to a control thread suitable for the current state to continue the execution of tasks.

For example, FIGS. 5A to 5D are examples of a sensor fusion switching method according to an embodiment of the present invention. The autonomous mobile vehicle V of this embodiment is, for example, an automatic pick-up and delivery vehicle equipped with a transfer machine for delivering goods from a warehouse to outdoor guests.

Please refer to Figure 5A, the autonomous mobile vehicle V receives the task instruction to transport the object O and the data required to execute the task instruction, including the position of the object O on the shelf S and the identification code I of the object O (QR as shown in the figure) Code), and then proceed to the state analysis, determine that it is located next to the shelf S in the warehouse, and enter the execution state to pick up the goods at this time. Among them, the autonomous mobile vehicle V uses the camera C to photograph the identification code I of the object O located on the shelf S to identify the object O, and when it is confirmed that the object O is the cargo indicated by the task instruction, it uses the transfer machine A picks up the item O.

Referring to FIG. 5B, after the goods are picked up, the autonomous mobile vehicle V is switched from the execution state to the transportation state, and the route planning module is activated to plan the delivery route. Among them, since the state transition is triggered during the process of switching from the execution state to the transportation state, the autonomous mobile vehicle V will sequentially switch the sensing combination until the sensing combination switched to matches the on-site positioning system.

For example, Table 1 below shows the sensing combinations in this state transition. The autonomous mobile vehicle V will switch among these sensing combinations in order to select the available sensing combinations to continue executing task commands. Among them, the autonomous mobile vehicle V found that it could not be matched with the on-site positioning system after using the sensing set 1, and then switched to the sensing set 2, and found that the sensing set 2 can be matched with the on-site positioning system, so the sensing set can be directly selected. Test set 2 continues to execute task instructions.

1. WiFi, IMU, rotary encoder 2. BLE, IMU, rotary encoder 3. Optical sensor, IMU, rotary encoder Table 1

Referring to Figure 5C, when the autonomous mobile vehicle V moves according to the planned path and is about to move from the warehouse to the outdoors, the state mapped by the current estimated position is different from the state estimated at the previous point in time (ie , The work phase is changed from warehouse to outdoor), at this time, the state transition will be triggered again and the sensing combination will be rescheduled.

For example, Table 2 below shows the sensing combination in this state transition. The autonomous mobile vehicle V finds that it can be matched with the on-site positioning system after using the sensing set 1, so the sensing set 1 can be directly selected to continue to execute the task instructions. Among them, because the main mobile vehicle V is switched according to the sequence of the most likely matching sensing combination under this state transition (that is, the work phase is changed from the warehouse to the outdoor), it can efficiently and seamlessly switch the positioning system.

1. GPS, base station 2. BLE, IMU, rotary encoder 3. Optical sensor, IMU, rotary encoder Table 2

Referring to FIG. 5D, after the autonomous mobile vehicle V arrives at the unloading site, it can estimate the current state as the execution state by estimating the current position and mapping the estimated current position to the semantic level. Switching from the transportation state to the execution state will trigger the state transition. At this time, the autonomous mobile vehicle V will switch the sensing combination to perform the recognition operation that needs to be performed when unloading.

For example, Table 3 below shows the sensing combination in this state transition. When the autonomous mobile vehicle V switches to the sensing combination 1, the camera will be activated. Since the camera supports the recognition operation (such as face recognition) of the unloading object T that needs to be performed during unloading, the autonomous mobile vehicle V can directly select the sensing combination 1 Continue to execute task instructions. When it is confirmed that the identity of the unloading object T matches, the autonomous mobile vehicle V starts the transfer machine A to deliver the object O to the unloading object T.

1. Camera 2. GPS, base station 3. BLE, IMU, rotary encoder table 3

In summary, the mobile vehicle and its state estimation and sensing fusion switching method of the present invention establishes a semantic level by dividing task commands into multiple work phases and mapping them to different states. When loading and transporting objects, you can map the estimated position to the current state and determine whether a state transition occurs. When a state transition occurs, you can quickly switch to a sensing combination suitable for the current state to continue execution Task instructions. Thereby, the state estimation of the mobile vehicle and its sensing fusion switching can be performed efficiently, and seamless switching between positioning systems can be realized.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to those defined by the attached patent application scope.

10: mobile vehicle 12: Data acquisition device 14: Sensor 16: Actuator 18: storage device 20: processor 30: Semantic level 32: Map location layer 34: Work stage layer 36: State layer 40: State transition model A: Transfer machinery C: Camera I: identification code O: Object P1~P4: working stage S: Shelf T: Unload the object V: Autonomous mobile vehicle W: Warehouse S202~S208: steps

FIG. 1 is a block diagram of a mobile vehicle according to an embodiment of the present invention. FIG. 2 is a flowchart of a method for fusion switching between state estimation and sensing of a mobile vehicle according to an embodiment of the present case. Fig. 3 is a schematic diagram of the semantic level drawn according to an embodiment of the present case. Fig. 4 is a schematic diagram of a state transition model drawn according to an embodiment of the present case. 5A to 5D are examples of a sensor fusion switching method according to an embodiment of the present application.

S202~S208: steps

Claims (16)

A method for switching between state estimation and sensing fusion of a mobile vehicle. The mobile vehicle includes at least one sensor, at least an actuator, and a processor for transferring and transporting objects. The method includes the following steps: Receiving a task instruction for moving the object and the data required to execute the task instruction; Divide the task instruction into multiple work phases according to the mapping position, and map each of the work phases to one of the transportation state and the execution state to establish a semantic hierarchy; Using the sensor to estimate the current position of the mobile vehicle; and The current position is mapped to one of the working stages in the semantic level to estimate the current state of the mobile vehicle. The method described in item 1 of the scope of patent application, wherein the task instruction is divided into multiple work phases according to the mapping position, and each of the work phases is mapped to one of the transportation state and the execution state to establish a semantic level After the steps, the method further includes: According to the sequence and connection relationship between the work phases, each of the work phases is mapped to one of the transportation state and the execution state along with the semantic level to form a state transition model. The method described in item 2 of the scope of patent application, wherein after the step of estimating the current state of the mobile vehicle, the method further includes: Compare the current state with the previous state estimated at the previous point in time to determine whether a state transition has occurred; and When it is determined that the state transition occurs, the multiple sensing combinations under the state transition are sequentially switched according to the state transition model to select the available sensing combinations to continue to execute the task instruction, wherein each of the sensing combinations The sensing combination includes at least one of the sensor and the actuator. The method described in item 1 of the scope of patent application, wherein the task instruction is composed of at least one of loading, unloading, and transportation, and the task instruction is divided into multiple work stages according to the mapping position, and each of the tasks is mapped. The steps from the working stage to the transportation state and the execution state to establish the semantic level include: The tasks are respectively corresponding to at least one control thread, and the work phases are distinguished according to the control thread. According to the method described in item 3 of the scope of the patent application, the loading and the unloading include distinguishing the working stages according to the loading location, the unloading location, the transfer object, and the identification of the loading object and the unloading object. The method according to item 4 of the scope of patent application, wherein the method for identifying the loading object and the unloading object includes biological characteristics, object characteristics, environmental characteristics or identification codes. The method according to item 3 of the scope of patent application, wherein the transportation work includes distinguishing the work stages according to the respective geographic information systems of at least one place through which the transportation passes. As the method described in item 1 of the scope of patent application, it also includes: Using the sensor to detect obstacles on the transport path of the mobile carrier; and When the obstacle is detected, the transportation path of each working stage in the transportation state is re-planned. A mobile vehicle including: Data capture device; At least one sensor for estimating the current position of the mobile vehicle; At least the actuator is used to transfer and transport objects; A storage device that stores the data and multiple computer commands or programs captured by the data capture device; and The processor is coupled to the data acquisition device, the sensor, the actuator, and the storage device, and is configured to execute the computer command or program to: Using the data capture device to receive a task command for transporting the object and data required to execute the task command; Divide the task instruction into multiple work phases according to the mapping position, and map each of the work phases to one of the transportation state and the execution state to establish a semantic level; and The current position estimated by the sensor is mapped to one of the working stages in the semantic level to estimate the current state of the mobile vehicle. For the mobile vehicle described in item 9 of the scope of patent application, the processor further maps each of the working stages to the transportation along the semantic level according to the sequence and connection relationship between the working stages. One of the state and the execution state to form a state transition model. As for the mobile vehicle described in item 10 of the scope of patent application, the processor further compares the current state with the previous state estimated at the previous time point to determine whether a state transition has occurred, and when determining that the state transition occurs, During the state transition, the plurality of sensing combinations under the state transition are sequentially switched according to the state transition model to select the available sensing combinations to successively execute the task instructions, wherein each sensing combination includes the At least one of the sensor and the actuator. The mobile vehicle according to item 9 of the scope of patent application, wherein the task instruction is composed of at least one task of loading, unloading and transportation, and the processor corresponds to at least one control thread, respectively, And distinguish the working stages according to the control thread. As for the mobile vehicle described in item 12 of the scope of patent application, the loading and unloading include distinguishing the working stages according to the loading location, the unloading location, the transferred object, and the identification of the loading object and the unloading object. According to the mobile vehicle described in item 13 of the scope of patent application, the method for identifying the loading object and the unloading object includes biological characteristics, object characteristics, environmental characteristics or identification codes. According to the mobile vehicle described in item 12 of the scope of the patent application, the transportation work includes distinguishing the work stages according to the respective geographic information systems of at least one place through which the transportation passes. The mobile vehicle as described in claim 9, wherein the processor further uses the sensor to detect obstacles on the transport path of the mobile vehicle, and when the obstacle is detected Re-planning the transportation path of each working stage in the transportation state.

Images