TW202012884A - Route-teaching module of movable carrier that comprises a visual sensing element and a receiver, a computing device, and a storage - Google Patents

Abstract

Disclosed is a route-teaching module of a movable carrier. The movable carrier can be a movable robot or a vehicle. The module comprises a visual sensing element and a receiver, a computing device, and a storage. The receiver receives an instruction command issued from a remote site to teach the movable carrier to move along an indicated route. The visual sensing element functions for photographing an image, and photographs successively the images of the route that the movable carrier moves along, and transmits the data of the successively photographed images to the computing device. The computing device calculates the route to provide an instructed route to the storage. The storage provides the instructed route to the movable carrier, so that the movable carrier may make movement.

Description

Teaching path module of mobile vehicle

The present invention, as a teaching path module of a mobile vehicle, refers to a module that can be installed on a mobile vehicle. The mobile vehicle includes a mobile robot or a car body. The module is used as a mobile vehicle to move to a destination Guidance module of the path of the position. The data calculation method of the module is to calculate the feature points using the continuously captured images, and then use the feature points to calculate the walking path to the destination, thereby providing the displacement of the moving vehicle Direction path.

Existing in factories, golf courses, restaurants, etc., there is a need to use mobile vehicles to replace humans to carry goods or patrol and monitor work. At present, the walking path modules of mobile vehicles used in these areas are equipped with sensing elements. There are multiple sensors or infrared or QR Cod corresponding components on the key point of the path, so as to provide information matching between the sensing component and the corresponding component on the moving path of the moving vehicle to perform the fixed path displacement However, the corresponding elements of such a guide path must be set at all necessary turning positions, so the number of corresponding elements required is large. If one of the corresponding elements fails, the moving vehicle cannot move.

And, when these work areas encounter a fire, or the environment path of the work area changes, the corresponding element cannot be paired with the sensing element of the moving vehicle; and, these sensing elements must be set at a fixed position. If the position of the execution environment is changed, these sensing elements must be rebuilt at the corresponding positions.

GPS is also used for path navigation, but the positioning accuracy of GPS is about a few meters, and if the robot moves indoors, even shuttles between the inside and outside of the building or between floors, these GPS cannot accurately provide mobile vehicle movement and execution jobs.

To this end, this work is to improve the shortcomings of conventional knowledge, use photography to obtain images to create feature point images, to obtain navigation paths, and use image feature points to calculate guidance paths, without the need for powerful calculations and large storage, which can meet the needs of fast calculations .

The invention uses visual images to establish the data of the teaching path. The stored data is a fixed path generated by the feature points of the visual image, and only the path data generated by the feature points is stored for the purpose of minimizing the amount of data information.

The position confirmation module of the mobile vehicle of the present invention adopts a way of guiding the path to locate the destination, which is convenient for fast storage and quick comparison after reading; and, provides an arithmetic unit to quickly compare the old data in the storage Yes, to update the memory to generate the latest path update.

The teaching path module of the mobile vehicle of the present invention is provided with a driving device and a controller. The teaching path module is connected to the controller assembled on the mobile vehicle by circuit or wireless transmission; The teaching path module includes: a visual sensing element, a receiver and an arithmetic unit, and a memory. The receiver receives the message of the teaching path sent by the remote end and provides the moving direction for the moving vehicle to move; And, the visual sensing element circuit is connected to an arithmetic unit, the arithmetic unit is connected to the storage, the storage system is used for storing and extracting the recording path data; and, the visual sensing element includes a camera lens; and, the The visual sensing element is disposed on any surface of the mobile vehicle; and the visual sensing element receives the receiver through the camera lens to guide the moving path of the mobile vehicle to continuously capture images, and provides multiple images It is transmitted to the arithmetic unit, which calculates the feature points obtained from multiple images to obtain the guidance path, and stores the guidance path to the storage. The specific process of this operation guidance path includes FAST, HARRIS, GFTT, SIFT, etc. Appropriate method, take out the 2D feature points in the photo by this method, and use the geometrical method such as Fundamental Matrix projection to calculate the position of the feature point in 3D using the parallax of the 2D feature point in different photos. Feature point and posture of the location of the mobile vehicle, so that the posture of the mobile vehicle and the two-dimensional feature point at that time and the three-dimensional feature point corresponding to the two-dimensional feature point, a corresponding relationship will be constructed, and the relationship path The data is stored in the storage together, which will be available for subsequent use; and, the storage only records the two-dimensional feature points generated by the image contour, the three-dimensional feature points and the path relationship data of the moving vehicle, so it is stored The data space is small, because the path relationship data is more intuitive and simple than the holographic image data, so when reloading, the current position of the mobile vehicle can be quickly compared, so that the mobile vehicle can quickly obtain the location, and therefore according to Guide the path positioning and be taught to take the next walking path to reach the destination; in addition, the guide path data creation requires less storage space in the memory, so the reading of the memory is fast, so the comparison is carried out now. It can have the fastest search guidance path and more accurate positioning accuracy when positioning.

1‧‧‧Visual sensing element

2‧‧‧Receiver

3‧‧‧Calculator

4‧‧‧storage

P1, P2‧‧‧Feature points

Pi‧‧‧3D feature points

O1, O2‧‧‧ pose

The first picture is the operation process of the created module.

The second picture is a block diagram of the implementation of this creation.

The third picture is the corresponding map of the characteristic points of the creation.

For the teaching path module of the mobile vehicle of the present invention, please refer to the first figure. The mobile vehicle A includes a mobile robot or a transport tool. The teaching path module B is assembled in the mobile vehicle A Built-in electronic components, which provide teaching function components of the position recognition and posture of the mobile vehicle A. The teaching path module B includes a visual sensing element 1 and a receiver 2 and an arithmetic unit 3 and a storage 4, The visual sensing element 1 electronic circuit connects the receiver 2 and the arithmetic unit 3, and the arithmetic circuit 3 electronic circuit is connected to the storage 4; and, the connection method may be direct circuit connection to the circuit board circuit, or network Wireless connection, or other communication connection; the implementation of wireless connection means that the arithmetic unit 3 can be located outside the module. As shown in the second figure, the arithmetic unit 3 is located on a control host C, by By wireless transmission or network cloud, the computing path information of the computing unit 3 is transmitted to the receiver 2, and then the receiver 2 stores the path information to the storage 4, whereby the storage 4 provides access path information, When the mobile vehicle wants to move, the receiver 2 obtains the storage path information from the storage 4 to provide the mobile vehicle with the predetermined path information for the mobile vehicle B to move according to the designated path; and, the visual sensing The element 1 includes a lens, which is installed at any surface position of the moving vehicle A as a scene function for shooting the environment where the vehicle is located. The photographic lens continuously shoots several images and transmits the resulting images to The arithmetic unit 3, the arithmetic unit 3 extracts discrete two-dimensional feature points from the two-dimensional image captured by the visual sensing element 1, and finds out that the same spatial point is different according to the characteristics of the continuously acquired image The two-dimensional coordinates on the image, due to the parallax of the moving position image of the moving vehicle, the two-dimensional coordinates of these two-dimensional feature points on each image will change accordingly, and use the continuously obtained two-dimensional feature point data to use The projection method calculates the true three-dimensional position of the feature point in space and the posture of the mobile vehicle itself. The posture refers to the direction that the mobile vehicle faces in three-dimensional space. The commonly used mathematical expression direction is Euler angle , Three-dimensional orthogonal matrix, unit quaternion; and, when the camera module of the visual sensing element is working, it will continue to take images of the current environment of the mobile vehicle, and pass the shooting data to the arithmetic unit, and repeat the above The method calculates new two-dimensional feature points and three-dimensional feature points and the new pose of the moving vehicle, and stores these data and relationship information into the storage for updating without rebuilding the entire storage 4.

The calculator 3 analyzes that if there are a large number of similar or overlapping feature points near a location, after comparing the data connected to the storage, it will automatically compare some similar or overlapping feature points or posture relations and other information Delete from the memory to reduce the data to occupy the memory space.

The visual sensing element can obtain image data and transmit it to the arithmetic unit through circuits, networks, mobile communications and the like.

The above operator 3 obtains image information, including: feature point description, two-dimensional coordinates of the feature point in the image, three-dimensional coordinates of the feature point in space, the position the mobile vehicle has traveled and the posture at that time, etc. The information is combined into a path file and pre-stored in the storage 4. When the moving vehicle A pauses or restarts or loses its position, the arithmetic unit 3 can compare the two-dimensional features extracted from the current captured image according to the content of the storage 4 Click to calculate the current path position of the mobile vehicle A, and send the path position information to the driving device of the mobile vehicle A to continue its movement.

Moreover, the visual sensing element 1 and the receiver 2 and the arithmetic unit 3 and the storage 4 may be more than one.

In addition, the storage 4 is provided in the teaching path module B, and can also be provided in the network cloud or externally connected to the memory of a computer host; and, the data of the storage 4 is provided to be directly used by the mobile vehicle A, or transmitted When it is used by another mobile vehicle A, the storage 4 can provide files for transfer to another storage body for storage.

The calculation method of the arithmetic unit 3 implementing the module of the present invention is as described in the following example. First, the optical imaging of the camera will more or less cause aberrations, and these phase differences will cause calculation errors in the spatial position; or even let the calculation results diverge; Therefore, before calculating the position, the arithmetic unit 3 must correct the aberration of the image according to the optical characteristics of the camera (for example: the pixel size of the image sensor, the pixel aspect ratio, the paraxial focal length of the optical lens, the difference between the vertical and horizontal focal lengths, distortion, etc.) To restore the true projection position of the feature point on the image.

Continue to implement the calculation and matching method. At the beginning, the user operates the mobile vehicle at the near end to move. According to the path to be traveled, the remote mobile vehicle walks the specified path once. When the mobile vehicle starts to move, the vision The sensing element 1 is activated, so continuous shooting is started, and the continuously shot images are provided to the arithmetic unit 3, the two-dimensional feature points are found in the arithmetic unit 3, and the two-dimensional feature points of each two images are found to be the same Feature point matching, using a fundamental matrix and multiple sets of feature point matching, usually 8 groups but not limited, to restore the three-dimensional position of the robot's pose and paired feature points, and use this geometric relationship to calculate all the feature points in the image Out of position in space.

Moreover, as shown in the second figure, the user can wirelessly send a control signal to the receiver 2 at the remote control host C, and the receiver 2 provides the control message to the built-in driving element of the mobile vehicle A (Not shown), and then the driving element drives the moving vehicle A to move according to the remote control, thereby moving the vehicle A to walk the specified path once, and when the moving vehicle begins to move, the visual sensing element 1 is Start up, so start continuous shooting, and provide the continuously shot images to the arithmetic unit 3, the arithmetic unit 3 calculates the image data into a path data file, and stores the path data file in the storage 4.

亦可簡化表示如下：[x,y,z,ψ,θ,φ] As shown in the third figure, from the many feature point pairs found in each of the two images I1, l2, when a feature point Pi in space is seen by the two images I1, l2, the camera center is in position Pose O1 and Pose O2, when the camera is in Pose O1, you can see the projection of the three-dimensional feature point Pi on the feature point p1; and, when the camera is in the Pose O2, the projection of the three-dimensional feature point Pi on the feature point p2, the second The projection of the center of the camera of the image l2 to the first image l1 is called the epipole e1. On the contrary, the pole of the pose O1 projected on l2 is e2, and any paired feature points P1 and P2 can be matched. It should be a three-dimensional feature point Pi in real space. The three-dimensional coordinate of the feature point is [x, y, z] t. The projection of multiple three-dimensional feature points can be used to calculate the pose O1 and pose O2 of the robot. The relationship is: O2= O1. [(R‧f)] Among them,

It can also be simplified as follows: [x,y,z,ψ,θ,φ]

Where x, y, z are the three-dimensional position coordinates, ψ , θ , φ represent the three rotation angles of yaw, pitch, and roll; to indicate the attitude of the vehicle in space, the order of operation is to turn yaw( ψ ) first, followed by pitch( θ ), finally roll( φ )

When two consecutive images are taken, any two of them can be matched with a certain number of feature points through the above methods, and the calculated spatial location of the feature points, as well as the relative position and posture of the module when shooting these two images. It can be continuously constructed as a structured data file and placed in the storage 4 to be stored in the form of a file, which can be reused in the future.

The usual usage is as follows: The visual sensing element 1 transmits the continuously captured scene image to the arithmetic unit 3, and the arithmetic unit 3 parses it into one or more two-dimensional feature points, usually at least 10 feature points, and then Compare the two-dimensional feature points with the two-dimensional feature points in one or more storages, so as to obtain the closest two-dimensional feature points stored in the storage, and obtain the position and posture of the vehicle at that time, It is used to initially locate the current pose, and then calculate and analyze the two-dimensional feature points extracted from the newly captured image and the two-dimensional feature points in the memory and the three-dimensional spatial position of the feature points in the memory, and can be moved. The current position of the path where the vehicle A is located. For this reason, the mobile vehicle A can continue to move based on this information. It can also send the guidance path file in the storage 4 to any mobile vehicle A for other The mobile vehicle A can reuse the file of this storage 4 to facilitate work, or upload the guidance path file in the storage 4, when other mobile vehicles A need to work in the same venue Send the guidance path file in the storage 4 remotely or proximally to the storage 4 in the teaching path module of another mobile vehicle A, thereby providing another mobile vehicle A to walk with the same guidance path .

Another implementation of this method is that it can be combined with other sensors as auxiliary posture data acquisition path position data sources, so as to integrate into a more reliable position estimation, for example: when the image features of the environment are scarce, the result calculated by the above method The error is large. If the path position data can be obtained from other sensors, it will help to correct the error of the image calculation posture; its advantages are:

1. In a specific area, a guidance path file can be created on the storage 4, the guidance path file is more than one file folder, or the file folder can be distributed and built on different storages 4.

2. The storage 4 can be stored as a guide path file, which can be directly used by other mobile vehicles A in the future, or transferred to another mobile vehicle A for use.

3. When the environment changes, the guidance path file in the storage 4 can be updated.

4. When the guidance file of the mobile vehicle A exceeds the original storage range of the storage 4, the storage 4 can be expanded.

5. When there are quite similar features or poses in the memory 4, after analysis, the arithmetic unit 4 will remove some feature point information from the memory 4.

6. The arithmetic unit 3 will calculate the feature point data of different feature changes caused by different time, light and shadow, environment, etc., and access the memory 4 to recalculate the feature point at any time.

In summary, these are only preferred embodiments of the present invention, and are not intended to limit the scope of implementation of the present invention, that is, all changes and modifications made in accordance with the patent application scope of the present invention are covered by the patent scope of the present invention.

1‧‧‧Visual sensing element

2‧‧‧Receiver

3‧‧‧Calculator

4‧‧‧storage

Claims (11)

A teaching path module for a mobile vehicle, the mobile vehicle includes a mobile robot or a transport tool, the teaching path module is assembled in a built-in electronic component of the mobile vehicle, the teaching path module includes a visual sense A measuring element and a receiver and a computing element and a storage, the visual sensing element electronic circuit connects the receiver and the computing element, the computing element electronic circuit connects to the memory; and, the visual sensing element includes a A lens, the photographic lens is installed at any surface position of the moving vehicle, and serves as a scene function for shooting the environment where the vehicle is located, and transmits the obtained image to the arithmetic unit, and the arithmetic unit receives the visual sensing element In the two-dimensional images taken, discrete two-dimensional feature points are extracted, and the two-dimensional coordinates of the same spatial point on different images are found according to the characteristics of the continuously obtained images, due to the image generated at the moving position of the moving vehicle Parallax, the two-dimensional coordinates of the two-dimensional feature points of the moving vehicle in the moving position image on each image will change accordingly, and the continuous acquisition of the two-dimensional feature point data is used to calculate the feature point in the space by projection The actual three-dimensional space position and the posture of the mobile vehicle itself, the posture refers to the direction that the mobile vehicle faces in three-dimensional space, and the commonly used mathematical expressions include Euler angle, three-dimensional orthogonal matrix, and unit quaternion And the camera module of the visual sensing element will continue to take images of the current environment of the mobile vehicle, and pass the shooting data to the arithmetic unit, and repeatedly calculate new 2D feature points and 3D feature points according to the above method New position and pose with the mobile vehicle, and store these data and relationship information in the storage to update, without the need to re-create the entire storage; and, the receiver is a remote control host Wirelessly send a control signal to the receiver, the receiver provides the control message to the built-in driving element of the mobile vehicle, and then the driving element drives the mobile vehicle to move according to the remote control, thereby the mobile vehicle walks the designated path , From the moment when the mobile vehicle starts to start, the visual sensing element is activated, so continuous shooting is started, and the continuously shot images are provided to the arithmetic unit, and the arithmetic unit calculates the path of the image data shot by the specified path Data file, and save the path data file to the storage. As the teaching path module of the mobile vehicle described in item 1 of the patent application scope, the connection method of the receiver and the arithmetic unit may be a direct circuit connection to a circuit on the circuit board, or a wireless network connection, or other communication connection; The implementation of the wireless connection means that the arithmetic unit can be a control host located outside the module, by wireless transmission or network cloud, the calculation path information of the arithmetic unit is transmitted to the receiver, and then the The receiver stores the path information to a storage, whereby the storage provides access path information, and when the mobile vehicle wants to move, the receiver obtains the storage through the storage to store the predetermined path information and provides it to the mobile vehicle For the mobile vehicle to move according to the specified path. As the teaching path module of the mobile vehicle described in item 1 of the patent application scope, more than one guidance path file can be created on the storage in a specific area of the storage, or the file folder can be distributed and built in different On the storage. As the teaching path module of the mobile vehicle described in item 1 of the patent scope, the storage can store multiple guidance path files, which can be used directly by other mobile vehicles in the future, or transferred to another mobile vehicle for use . According to the teaching path module of the mobile vehicle described in item 1 of the patent application scope, the guidance path file in the storage can be updated when the storage environment changes. According to the teaching path module of the mobile vehicle described in item 1 of the patent application scope, the storage can be expanded when the mobile vehicle guidance path file exceeds the storage storage range of the original storage. According to the teaching path module of the mobile vehicle as described in item 1 of the patent application scope, when the memory has quite similar features or poses in the memory, after analysis, the operator will transfer some feature point information from Remove from the storage. As the teaching path module of the mobile vehicle described in item 1 of the patent application scope, the arithmetic unit will calculate the characteristic point data of different characteristic changes caused by different time, light and shadow, environment, etc., and it will be calculated from the storage at any time Perform access to recalculate feature points. According to the teaching path module of the mobile vehicle described in item 1 of the patent application scope, there may be more than one visual sensing element, the receiver, the arithmetic unit, and the memory. A teaching path module for a mobile vehicle, the mobile vehicle includes a mobile robot or a transport tool, the teaching path module is assembled in a built-in electronic component of the mobile vehicle, the teaching path module includes a visual sense A measuring element and an arithmetic unit and a storage, the visual sensing element electronic circuit is connected to the receiver and the arithmetic unit, the arithmetic electronic circuit is connected to the storage; and, the visual sensing element includes a lens and the camera The lens is installed at any surface position of the moving vehicle, as a function of shooting the scenery of the environment in which the vehicle is located, and transmits the obtained image to the arithmetic unit, which calculates the two-dimensional image taken from the visual sensing element In the image, the discrete two-dimensional feature points are extracted, and the two-dimensional coordinates of the same spatial point on different images are found according to the characteristics of the continuously obtained image. Due to the parallax of the moving position image of the moving vehicle, these two The two-dimensional coordinates of the one-dimensional feature point on each image will change accordingly, and the continuous three-dimensional feature point data is used to calculate the true three-dimensional space position of the feature point in space and the moving vehicle itself by using the projection method. Posture, the posture refers to the direction that the moving vehicle faces in three-dimensional space, and the commonly used mathematical expressions include Euler angle, three-dimensional orthogonal matrix, and unit quaternion; and the camera model of the visual sensing element The team will continue to take images of the current environment of the mobile vehicle, and pass the shooting data to the calculator, and repeatedly calculate the new 2D feature points and 3D feature points and the new pose of the mobile vehicle according to the above method, and use these The data of the data and the relationship are stored and updated in the aforementioned storage without rebuilding the entire storage; the visual sensing element is activated from the moment the mobile vehicle starts to be activated, so continuous shooting is started and the continuous The captured image is provided to an arithmetic unit, and the arithmetic unit calculates a path data file of the image data captured by the specified path, and stores the path data file in the storage. The teaching path module of the mobile vehicle as described in item 10 of the patent application scope, the module is provided with a receiver, which connects the visual sensing element and the arithmetic unit; and, the receiver can be externally assembled in The driving component of the mobile vehicle; and, a remote control host receives wirelessly sending the control signal to the receiver, the receiver provides the control message to the built-in driving component of the mobile vehicle, and then the driving component drives the movement The vehicle moves according to the remote control, so that the moving vehicle is remotely walked to the designated path, and the image captured by the visual sensing element is transmitted to the arithmetic unit.

Images