CN111860050B

CN111860050B - Loop detection method and device based on image frames and vehicle-mounted terminal

Info

Publication number: CN111860050B
Application number: CN201910346794.XA
Authority: CN
Inventors: 李天威; 徐抗; 童哲航; 刘一龙; 谢国富
Original assignee: Beijing Chusudu Technology Co ltd
Current assignee: Momenta Suzhou Technology Co Ltd
Priority date: 2019-04-27
Filing date: 2019-04-27
Publication date: 2024-07-02
Anticipated expiration: 2039-04-27
Also published as: CN111860050A

Abstract

The embodiment of the invention discloses a loop detection method and device based on image frames and a vehicle-mounted terminal. The method comprises the following steps: acquiring first image frames of a plurality of directions around the vehicle acquired at a first place by an image acquisition device; updating the pixel value of each pixel point according to the surrounding pixel points of each pixel point aiming at each pixel point in each first image frame, and taking all the updated pixel points in the first image frame as the characteristic information of the first image frame; and according to the characteristic information of each image frame, matching each first image frame with a plurality of image frames corresponding to a plurality of places in an image database, and determining a place pointed to the same place as the first place from the second places indicated by the first matching result. By applying the scheme provided by the embodiment of the invention, the accuracy of loop detection can be improved.

Description

Loop detection method and device based on image frames and vehicle-mounted terminal

Technical Field

The invention relates to the technical field of automatic driving, in particular to a loop detection method and device based on image frames and a vehicle-mounted terminal.

Background

The loop detection means that when a movable object provided with a camera moves to a certain position, the collected picture is matched with a history picture, and when the matching is successful, the movable object is considered to move to the position corresponding to the history picture which is successfully matched. For example, the movable object may be an object such as a vehicle, a robot, or the like. The loop detection can correlate the location corresponding to the current image with the location corresponding to the history image.

However, when the vehicle moves to a certain position again, the angle of the image photographed by the camera on the vehicle may change, and when the image is matched with the history image, the matching may not be successful. When matching an image with a history image, matching is usually performed according to key points in the image, but when the light and shooting angle of the external environment change, the key points in the image change correspondingly, and matching according to the key points may not be successful. Based on the analysis, the accuracy of the loop detection is not high enough.

Disclosure of Invention

The invention provides a loop detection method and device based on image frames and a vehicle-mounted terminal, so as to improve accuracy in loop detection. The specific technical scheme is as follows.

In a first aspect, an embodiment of the present invention discloses a loop detection method based on image frames, including:

Acquiring first image frames of a plurality of directions around the vehicle acquired at a first place by an image acquisition device;

updating pixel values of the pixel points according to surrounding pixel points of each pixel point for each pixel point in each first image frame, and taking all updated pixel points in the first image frame as characteristic information of the first image frame;

According to the characteristic information of each image frame, matching each first image frame with a plurality of image frames corresponding to a plurality of places in an image database; when at least a preset number of successfully matched image frames exist in a plurality of image frames corresponding to the second location in the image database, determining to obtain a first matching result meeting preset location matching conditions; the location indicated by the first matching result is the second location; the image database is used for storing characteristic information of image frames at a plurality of first-type places, wherein the image frames at each first-type place comprise image frames in a plurality of directions around a vehicle at the place;

a location is determined from the second location that points to the same location as the first location.

Optionally, the image database is further configured to store feature information of image frames at a plurality of second class locations, where the image frames at each second class location include image frames in a direction around the vehicle at the location; after obtaining the feature information of the first image frame, the method further includes:

According to the characteristic information of each image frame, matching each first image frame with one image frame at each place in the image database; when at least one first image frame exists in each first image frame and is successfully matched with the image frames in the image database, determining to obtain a second matching result meeting a preset place matching condition; wherein, the third place indicated by the second matching result is: the image database is used for matching the corresponding place of the successfully matched image frame;

a location is determined from the third location that points to the same location as the first location.

Optionally, after acquiring each first image frame acquired by the image acquisition device at the first location, the method further includes:

splicing the first image frames to obtain a first panoramic image frame at the first place;

Updating pixel values of the pixel points according to surrounding pixel points of each pixel point for each pixel point in each first panoramic image frame, and taking all updated pixel points in the first panoramic image frame as characteristic information of the first panoramic image frame;

Matching the first panoramic image frame with the panoramic image frames corresponding to a plurality of places in an image database according to the characteristic information of each panoramic image frame; when the first panoramic image frame is successfully matched with the panoramic image frames in the image database, determining a third matching result meeting a preset place matching condition, wherein a fourth place indicated by the third matching result is: the image database is used for matching the corresponding place of the panoramic image frame successfully; the image database is used for storing characteristic information of panoramic image frames at a plurality of places;

a location is determined from the fourth location that points to the same location as the first location.

Optionally, when the second location is plural, the step of determining a location pointing to the same location as the first location from the second location includes:

acquiring a fifth location indicated by a matching result determined from the image database at a previous location before the first location;

Judging whether target image frames which are continuous with the image frames corresponding to the fifth location exist in the image frames corresponding to the second locations in the image database;

if so, a second location corresponding to the target image frame is determined as a location pointing to the same location as the first location.

Optionally, the step of matching each first image frame with a plurality of image frames corresponding to each location in the image database according to the feature information of each image frame includes:

Calculating, for each first image frame and each image frame in an image database, an absolute value of a difference between a pixel value of each pixel point in the first image frame and a pixel value of a pixel point at the same position in the image frame, and taking a sum of absolute values of differences between pixel values of two corresponding pixel points between the first image frame and the image frame as a feature information similarity between the first image frame and the image frame; and when the similarity of the characteristic information is smaller than a preset similarity threshold, determining that the first image frame is successfully matched with the image frame.

Optionally, when it is determined that a matching result satisfying the preset location matching condition cannot be obtained, the method further includes:

And storing the characteristic information of each first image frame at the first location into an image database.

Optionally, the step of updating, for each pixel point in each first image frame, the pixel value of the pixel point according to surrounding pixel points of the pixel point includes:

for each pixel point in each first image frame, updating the pixel value of the pixel point according to the following formula:

P’＝(P-Pμ)/Pσ

Wherein, P' is the updated pixel value of the pixel, P is the pixel value of the pixel, pμ is the average value of the pixel values of the surrounding pixels of the pixel, and pσ is the standard deviation of the pixel values of the surrounding pixels of the pixel.

Optionally, the step of acquiring a first image frame of a plurality of directions around the vehicle acquired by the image acquisition device at a first location includes:

and acquiring initial image frames of the plurality of directions around the vehicle, which are acquired by the image acquisition equipment at a first place, and performing downsampling processing on each initial image frame to obtain each first image frame.

In a second aspect, an embodiment of the present invention discloses a loop detection device based on image frames, including:

an acquisition module configured to acquire first image frames of a plurality of directions around the vehicle acquired by an image acquisition apparatus at a first location;

The first updating module is configured to update pixel values of the pixel points according to surrounding pixel points of each pixel point for each pixel point in each first image frame, and take all updated pixel points in the first image frame as characteristic information of the first image frame;

the first matching module is configured to match each first image frame with a plurality of image frames corresponding to a plurality of places in the image database according to the characteristic information of each image frame; when at least a preset number of successfully matched image frames exist in a plurality of image frames corresponding to the second location in the image database, determining to obtain a first matching result meeting preset location matching conditions; the location indicated by the first matching result is the second location; the image database is used for storing characteristic information of image frames at a plurality of first-type places, wherein the image frames at each first-type place comprise image frames in a plurality of directions around a vehicle at the place;

A first determination module configured to determine a location from the second locations that is pointed to the same location as the first location.

Optionally, the image database is further configured to store feature information of image frames at a plurality of second class locations, where the image frames at each second class location include image frames in a direction around the vehicle at the location; the apparatus further comprises:

The second matching module is configured to match each first image frame with one image frame at each place in the image database according to the characteristic information of each image frame after the characteristic information of the first image frame is obtained; when at least one first image frame exists in each first image frame and is successfully matched with the image frames in the image database, determining to obtain a second matching result meeting a preset place matching condition; wherein, the third place indicated by the second matching result is: the image database is used for matching the corresponding place of the successfully matched image frame;

A second determination module configured to determine a location from the third location that points to the same location as the first location.

Optionally, the apparatus further includes:

a stitching module configured to stitch each first image frame after acquiring each first image frame acquired by the image acquisition device at the first location, so as to obtain a first panoramic image frame at the first location;

The second updating module is configured to update pixel values of the pixel points according to surrounding pixel points of each pixel point for each pixel point in each first panoramic image frame, and take all updated pixel points in the first panoramic image frame as characteristic information of the first panoramic image frame;

The third matching module is configured to match the first panoramic image frame with the panoramic image frames corresponding to a plurality of places in the image database according to the characteristic information of each panoramic image frame; when the first panoramic image frame is successfully matched with the panoramic image frames in the image database, determining a third matching result meeting a preset place matching condition, wherein a fourth place indicated by the third matching result is: the image database is used for matching the corresponding place of the panoramic image frame successfully; the image database is used for storing characteristic information of panoramic image frames at a plurality of places;

A third determination module configured to determine a location from the fourth location that points to the same location as the first location.

Optionally, when the second location is plural, the first determining module is specifically configured to:

Optionally, the first matching module is specifically configured to:

Optionally, the apparatus further includes:

and the storage module is configured to store the characteristic information of each first image frame at the first location into the image database when the matching result meeting the preset location matching condition cannot be obtained.

Optionally, the first updating module is specifically configured to:

P’＝(P-Pμ)/Pσ

Optionally, the acquiring module is specifically configured to:

In a third aspect, an embodiment of the present invention discloses a vehicle-mounted terminal, including: a processor and an image acquisition device; the processor includes: the device comprises an acquisition module, a first updating module, a first matching module and a first determining module;

the acquisition module is used for acquiring first image frames of a plurality of directions around the vehicle, acquired by the image acquisition equipment at a first place;

The first updating module is configured to update, for each pixel point in each first image frame, a pixel value of the pixel point according to surrounding pixel points of the pixel point, and take all updated pixel points in the first image frame as feature information of the first image frame;

The first matching module is used for matching each first image frame with a plurality of image frames corresponding to a plurality of places in the image database according to the characteristic information of each image frame; when at least a preset number of successfully matched image frames exist in a plurality of image frames corresponding to the second location in the image database, determining to obtain a first matching result meeting preset location matching conditions; the location indicated by the first matching result is the second location; the image database is used for storing characteristic information of image frames at a plurality of first-type places, wherein the image frames at each first-type place comprise image frames in a plurality of directions around a vehicle at the place;

the first determining module is used for determining a place which points to the same place as the first place from the second place.

Optionally, the image database is further configured to store feature information of image frames at a plurality of second class locations, where the image frames at each second class location include image frames in a direction around the vehicle at the location; the processor further includes:

The second matching module is used for respectively matching each first image frame with one image frame at each place in the image database according to the characteristic information of each image frame after the characteristic information of the first image frame is obtained; when at least one first image frame exists in each first image frame and is successfully matched with the image frames in the image database, determining to obtain a second matching result meeting a preset place matching condition; wherein, the third place indicated by the second matching result is: the image database is used for matching the corresponding place of the successfully matched image frame;

And the second determining module is used for determining a place which points to the same place as the first place from the third place.

Optionally, the processor further includes:

the splicing module is used for splicing the first image frames after acquiring the first image frames acquired by the image acquisition equipment at the first place to obtain a first panoramic image frame at the first place;

The second updating module is used for updating the pixel values of the pixel points according to the surrounding pixel points of each pixel point for each pixel point in each first panoramic image frame, and taking all the updated pixel points in the first panoramic image frame as the characteristic information of the first panoramic image frame;

The third matching module is used for matching the first panoramic image frame with the panoramic image frames corresponding to a plurality of places in the image database according to the characteristic information of each panoramic image frame; when the first panoramic image frame is successfully matched with the panoramic image frames in the image database, determining a third matching result meeting a preset place matching condition, wherein a fourth place indicated by the third matching result is: the image database is used for matching the corresponding place of the panoramic image frame successfully; the image database is used for storing characteristic information of panoramic image frames at a plurality of places;

And a third determining module, configured to determine a location pointing to the same location as the first location from the fourth location.

Optionally, when the second location is multiple, the first determining module is specifically configured to:

Optionally, the first matching module is specifically configured to:

Optionally, the processor further includes:

And the storage module is used for storing the characteristic information of each first image frame at the first location to the image database when the matching result meeting the preset location matching condition cannot be obtained.

Optionally, the first updating module is specifically configured to:

P’＝(P-Pμ)/Pσ

Optionally, the acquiring module is specifically configured to:

As can be seen from the above, the loop detection method and device based on image frames and the vehicle-mounted terminal provided by the embodiments of the present invention can acquire the first image frames in multiple directions around the vehicle acquired at the first location, and match the multiple image frames between every two locations when matching with the image frames in the image database, so that loop detection can be accurately realized even if the shooting angle changes during re-shooting, and robustness of loop detection in terms of image shooting angle is improved; the pixel points in the first image frame are updated according to the pixel values of surrounding pixel points, even if the environmental relationship changes during secondary shooting, the matched image frame can be accurately detected, and the robustness of the image to light changes is improved; the global features of all pixel points of the first image frame are adopted as feature information, so that the accuracy of loop detection can be improved. Therefore, the embodiment of the invention can improve the accuracy of loop detection.

The innovation points of the embodiment of the invention include:

1. The obtained image frames are subjected to pixel value updating, all pixel points of the image frames are adopted as feature information, and the global feature and the updating processing of the pixel values can improve the robustness of the image frames when the image frames are corresponding to light changes; and each place corresponds to the image frames in a plurality of directions, and when at least a preset number of groups of successfully matched image frames exist under the condition that each place contains the image frames in all directions in the image database, a matching result meeting the preset place matching condition is determined. This makes the matching in loop detection not limited by the shooting angle, and the accuracy of the same place obtained by matching is higher.

2. When at least one first image frame is successfully matched with the image frames in the image database under the condition that each place in the image database contains a certain image frame, a matching result meeting the preset place matching condition is determined. This enables loop detection to be compatible with the case where each location in the database corresponds to one image frame, improving compatibility.

3. And the plurality of image frames corresponding to each location are spliced into the full-scene image frames, so that the matching times can be reduced, the matching flow is shortened, and the matching efficiency is improved.

4. Based on the image sequence, the accuracy of loop detection can be improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is apparent that the drawings in the following description are only some embodiments of the invention. Other figures may be derived from these figures without inventive effort for a person of ordinary skill in the art.

Fig. 1 is a schematic flow chart of a loop detection method based on image frames according to an embodiment of the present invention;

Fig. 2 is a reference diagram before and after updating pixel values of an image frame according to an embodiment of the present invention;

fig. 3 is a flowchart of another loop detection method based on image frames according to an embodiment of the present invention;

Fig. 4 is a flowchart of another loop detection method based on image frames according to an embodiment of the present invention;

Fig. 5 to 7 are schematic structural diagrams of a loop detection device based on image frames according to several embodiments of the present invention;

Fig. 8 is a schematic structural diagram of a vehicle-mounted terminal according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without any inventive effort, are intended to be within the scope of the invention.

It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. A process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed but may alternatively include other steps or elements not listed or inherent to such process, method, article, or apparatus.

The embodiment of the invention discloses a loop detection method and device based on image frames and a vehicle-mounted terminal, which can improve the accuracy of loop detection. The following describes embodiments of the present invention in detail.

Fig. 1 is a schematic flow chart of a loop detection method based on image frames according to an embodiment of the present invention. The method is applied to the electronic equipment. The electronic equipment can be common computers, servers or intelligent mobile terminals and other equipment. The electronic device may also be a device for calculation processing installed in a vehicle. The method specifically comprises the following steps.

S110: first image frames of a plurality of directions around a vehicle acquired at a first location by an image acquisition device are acquired.

The image acquisition devices can be arranged at different positions of the vehicle respectively, and each image acquisition device acquires image frames in one direction around the vehicle. For example, one camera may be installed in the front, rear, left, and right directions of the vehicle, respectively, in which case 4 first image frames may be acquired at each location. The image acquisition device can be a common lens camera or a fisheye lens camera.

The electronic device may receive a first image frame acquired by each image acquisition device when the vehicle is located at a first location. At the same time and the same place, each image acquisition device acquires a first image frame. The electronic device may associate each first image frame with a first location. Wherein the first location may be represented in two or three dimensional coordinates.

The image capturing device may capture image frames at a preset frame rate. When loop detection is performed, the electronic device may acquire a key frame (I-frame) in an image frame sequence acquired by the image acquisition device as a first image frame. This can reduce the amount of calculation. In the positioning process, the movement of the vehicle is slower than the acquisition speed of the image acquisition equipment, so that loop detection is not required for each image frame, and the processing burden of the system can be reduced by selecting key frames in the image frames for loop detection. This also maintains the validity of the information.

In acquiring the first image frames of the plurality of directions around the vehicle acquired by the image acquisition device at the first location, specifically may include: initial image frames of the image acquisition equipment in multiple directions around the vehicle are acquired at a first place, and all the initial image frames are converted into gray images to obtain all the first image frames. The above embodiment is applicable to the case where the image directly acquired by the image acquisition apparatus is a color image.

S120: and updating the pixel value of each pixel point in each first image frame according to the surrounding pixel points of the pixel point, and taking all the updated pixel points in the first image frame as the characteristic information of the first image frame.

And updating each pixel point in each first image frame in the mode, so that the characteristic information of each first image frame can be obtained.

Specifically, for each pixel point in each first image frame, the pixel value of the pixel point may be updated according to the following formula:

P’＝(P-Pμ)/Pσ

Wherein P' is the updated pixel value of the pixel, P is the pixel value of the pixel, Pμ is the average value of the pixel values of the surrounding pixels of the pixel, and Pσ is the standard deviation of the pixel values of the surrounding pixels of the pixel. The surrounding pixels of the pixel may include 8 adjacent pixels around the pixel, which may also be referred to as eight neighboring pixels. The surrounding pixels may be other defined pixels. Pμ is the arithmetic average of the pixel values of surrounding pixels.

Where k is the total number of surrounding pixels of the pixel, and i is the number of each surrounding pixel.

After the pixel value of the pixel point is updated according to the surrounding pixel points of the pixel point, the robustness of the image frame to the light intensity during matching can be improved. According to experimental result data, after the pixel values of the pixel points are updated according to P' = (P-P mu)/P sigma, the robustness of the pixel points to light intensity during image frame matching can be effectively improved, and the image matching is more accurate.

Referring to fig. 2, fig. 2 is a reference diagram after updating the pixel points of the first image frame. The upper two images are two image frames in the underground garage, and after the pixel values of the pixel points of the two image frames are processed, the first image frames with the updated pixel values of the pixel points below can be obtained respectively.

In this embodiment, all updated pixel points in the first image frame are used as the feature information of the first image frame, so that it is known that the feature information of the first image frame adopts the global feature of the image frame, and compared with the feature information of the key points in the image frame as the feature information of the image frame, the global feature can have invariance to extreme appearance influence caused by day and night, weather, seasons and other reasons, so that the robustness to light intensity during image frame matching can be improved.

S130: and according to the characteristic information of each image frame, matching each first image frame with a plurality of image frames corresponding to a plurality of places in an image database.

Wherein the image database is for storing feature information of image frames at a plurality of first-type places, the image frames at each first-type place including image frames of a plurality of directions around a vehicle at the place.

And when at least a preset number of successfully matched image frames exist in a plurality of image frames corresponding to the second location in the image database, determining to obtain a first matching result meeting a preset location matching condition. The primary matching process between the first location and each first type location in the image database comprises a process of pairwise matching between each first image frame corresponding to the first location and each image frame corresponding to the first type location.

The location indicated by the first matching result is a second location, and the second location is a location in the first type of location. The second location may be represented in two or three dimensional coordinates.

The preset number may be a number determined in advance from the total number of first image frames contained in each first location. For example, when the total number of the first image frames is 4, the preset number may be a value of 2 to 4 or the like.

For example, the first location contains 4 first image frames, and each first type location in the image database contains 4 image frames. At the time of matching, the feature information of each first image frame may be matched with each image frame in each first-type place. That is, for each first location and each first type location, 4*4 total 8 image matches may be performed, and when there are more than a preset number of successfully matched image frames, the matching between the first location and the first type location is considered to satisfy a preset location matching condition, where the first type location is the second location. When the number of the matched image groups is smaller than the preset number of the groups, the matching between the first place and the first type of places is considered to not meet the preset place matching condition, namely, a matching result meeting the preset place matching condition does not exist.

When the matching result meeting the preset place matching condition cannot be obtained, the characteristic information of each first image frame at the first place can be stored in the image database for subsequent loop detection.

S140: a location is determined from the second location that points to the same location as the first location.

When the number of the second places is one, the second place may be directly determined as a place directed to the same place as the first place. When the number of the second places is greater than one, one place may be selected from the second places as a place directed to the same place as the first place, or all the second places may be determined as places directed to the same place as the first place.

A location pointing to the same location as the first location, it is understood that the location may be different from the first location in terms of coordinates, but both locations represent the same location.

After determining the location that points to the same location as the first location, the first location may also be modified based on the determined location.

In an application scenario, in some special places without global positioning system (Global Positioning System, GPS) signals, a vehicle cannot determine a specific position of the vehicle according to the GPS signals, and thus, the construction of a map cannot be completed. For example, in a place such as an underground garage, global position information of a vehicle cannot be obtained by means of a GPS signal. In this case, the current vehicle pose can only be determined by means of the odometer data and the previous vehicle pose. The accumulated error of the map constructed by the method is larger. To reduce the accumulated error, an image database may be constructed to store image frames acquired by the camera during vehicle travel for loop detection.

When the vehicle moves to the first place, if a first matching result meeting a preset place matching condition can be obtained from the image database, the first place can be corrected according to the determined place, and then the pose of the vehicle at the first place is corrected, so that the accumulated error in the positioning process is reduced.

As can be seen from the above, in this embodiment, the first image frames in multiple directions around the vehicle collected at the first location can be obtained, and when the first image frames are matched with the image frames in the image database, the multiple image frames between every two locations are matched, so that loop detection can be accurately realized even if the shooting angle changes during re-shooting, and the robustness of loop detection in terms of the image shooting angle can be improved; the pixel points in the first image frame are updated according to the pixel values of surrounding pixel points, even if the environmental relationship changes during secondary shooting, the matched image frame can be accurately detected, and the robustness of the image to light changes is improved; the global features of all pixel points of the first image frame are adopted as feature information, so that the accuracy of loop detection can be improved. Therefore, the accuracy of loop detection can be improved.

In another embodiment of the present invention, when there are a plurality of second sites based on the embodiment shown in fig. 1, step S140, the step of determining a site pointing to the same site as the first site from the second sites, may specifically include steps 1a to 3a.

Step 1a: a fifth location indicated by the result of the match determined from the image database at the previous location before the first location is obtained.

In each loop detection, a plurality of second locations matching the first location may be determined from the image database. Further, there are cases where the frame numbers of the image frames matched from the image database at consecutive points are consecutive. When the image frames employ key frames, consecutive frame numbers refer to consecutive key frame numbers. In order to more accurately determine a location from a plurality of second locations that points to the same location as the first location, the plurality of second locations may be screened based on consecutive locations, i.e. based on the image sequence.

The image acquisition device may acquire key frames having consecutive frame numbers while the vehicle is in continuous motion. At the same place, the frame numbers of the first image frames acquired by the image acquisition devices on the same vehicle are the same.

Wherein the last place can be one or more. I.e. the matching result at one or more places before the first place can be obtained.

Step 2a: judging whether target image frames which are continuous with the image frames corresponding to the fifth location exist in the image frames corresponding to a plurality of second locations in the image database, and if so, executing the step 3a; if not, it may not be processed.

The image frame corresponding to the fifth location refers to an image frame corresponding to a location pointing to the same location in the image database corresponding to the fifth location. The fifth location may be one or more.

For example, the current location is location 3, the vehicle moves in the order of location 1, location 2, and location 3, and location 1, location 2, and location 3 are three consecutive locations. At location 3, the location(s) determined from the image database to point to the same location as location 3 include: site a-image frame a, site b-image frame b, site c-image frame c. The previous location may include location 1 and location 2 relative to location 3. When the vehicle is at the place 2, determining that the place pointed to the same place as the place 2 from an image database is a place m, and determining that an image frame number corresponding to the place m in the image database is m; when the vehicle is at the point 1, the point which is pointed to the same point as the point 1 and is determined from the image database is the point n, and the image frame number corresponding to the point n in the image database is n. It is determined that the image frame corresponding to the location 3 (second location) includes the image frame a, the image frame b, and the image frame c, wherein the frame numbers of the image frame a, the image frame m, and the image frame n are consecutive, and the image frame a can be determined as the target image frame.

In loop detection for each location, if there is a continuous phenomenon in frame numbers of image frames in the image database corresponding to consecutive locations, the location in loop detection can be determined more accurately.

Step 3a: a second location corresponding to the target image frame is determined as a location pointing to the same location as the first location.

In summary, the present embodiment determines a location pointing to the same location as the first location based on the image sequence. Compared with matching based on single frames, the embodiment can avoid error matching caused by single frame image content approximation at different places, and can improve the accuracy of loop detection.

In another embodiment of the present invention, based on the embodiment shown in fig. 1, step S130, according to the feature information of each image frame, may specifically include the step of matching each first image frame with a plurality of image frames corresponding to each location in the image database, where:

for each first image frame and each image frame in the image database, calculating an absolute value of a difference between a pixel value of each pixel point in the first image frame and a pixel value of a pixel point at the same position in the image frame, and taking a sum value of absolute values of differences between pixel values of two corresponding pixel points between the first image frame and the image frame as a feature information similarity between the first image frame and the image frame.

And when the similarity of the characteristic information is smaller than a preset similarity threshold, determining that the first image frame is successfully matched with the image frame. And when the similarity of the characteristic information is not smaller than a preset similarity threshold value, determining that the first image frame fails to be matched with the image frame.

The preset similarity threshold may be a value set empirically in advance.

A specific example is described below on how to determine the feature information similarity between two image frames. For example, the first image frame a and the image frame B need to be matched, and it is known that the first image frame a and the image frame B are 60px×80px images, i.e. 4800 pixels in each image frame. And respectively differencing the pixel points at the same position of the first image frame A and the image frame B to obtain 4800 difference values. And summing the absolute values of the 4800 differences to obtain the similarity of the characteristic information between the first image frame A and the image frame B.

The more similar the first image frame is to the image frame, the less similar the resulting feature information is. Therefore, when the feature information similarity is smaller than the preset similarity threshold, the first image frame can be considered to be successfully matched with the image frame.

In another embodiment, a SAD (pixel by pixel Sum of Absolute Differences) algorithm may be used to determine the feature information similarity between the first image frame and the image frame.

In summary, the embodiment provides a specific implementation manner when each first image frame is matched with each image frame in the image database, so that matching efficiency can be improved.

In another embodiment of the present invention, based on the embodiment shown in fig. 1, in order to reduce the complexity of calculation and improve the processing efficiency, step S110, the step of acquiring the first image frames of multiple directions around the vehicle acquired by the image acquisition device at the first location may specifically include:

Initial image frames of the image acquisition equipment in multiple directions around the vehicle are acquired at a first place, and downsampling processing is carried out on all the initial image frames to obtain all the first image frames.

The down-sampling processing for each initial image frame may specifically include: and downsampling each initial image frame according to a preset first target resolution. Each of the obtained first image frames is a target resolution. The first target resolution is less than the resolution of the initial image frame.

The down-sampling process for each initial image frame may also include: and determining a second target resolution of the image frames after downsampling according to the resolutions of the initial image frames, and downsampling each initial image frame to the image frames with the second target resolution to obtain each first image frame. The second target resolution is less than the resolution of the initial image frame.

For example, when the resolution of the initial image frame is m×n pixels and the initial image frame is downsampled by s times, a first image frame with resolution of (M/s) ×n/s may be obtained, where the image in the s×s window in the initial image frame will become a pixel, and this pixel is the average of all the pixels in the window.

In another embodiment, before the down-sampling process is performed on each initial image frame, the de-distortion process may be further performed on each initial image frame, and the down-sampling process may be performed on each initial image frame after the de-distortion process. The de-distortion can improve the accuracy of the characteristic information of the image frames, and further improve the accuracy of the image frames during matching.

In summary, in this embodiment, the downsampling process is performed on the acquired initial image frames to obtain each first image frame, which can reduce the resolution of the image frames, reduce the feature information amount of the image frames, further reduce the calculation amount when the image frames are matched, and improve the calculation efficiency.

In another embodiment of the invention, the embodiment shown in fig. 3 can be obtained on the basis of the embodiment shown in fig. 1. In the embodiment shown in fig. 3, the image database is further configured to store feature information of image frames at a plurality of locations of a second type, each image frame at a location of the second type comprising image frames of a direction around the vehicle at the location.

S310: first image frames of a plurality of directions around a vehicle acquired at a first location by an image acquisition device are acquired.

S320: and updating the pixel value of each pixel point in each first image frame according to the surrounding pixel points of the pixel point, and taking all the updated pixel points in the first image frame as the characteristic information of the first image frame.

In this embodiment, the steps S310 and S320 are the same as the steps S110 and S120, respectively, in the embodiment shown in fig. 1, and the detailed description will be made with reference to the embodiment shown in fig. 1.

S330: and matching each first image frame with one image frame at each place in the image database according to the characteristic information of each image frame. And when at least one first image frame exists in each first image frame and is successfully matched with the image frames in the image database, determining to obtain a second matching result meeting the preset place matching condition.

The third location indicated by the second matching result is: and the image database is used for matching the corresponding place of the successfully matched image frame.

In this embodiment, each location in the image database corresponds to an image frame. And when at least one first image frame is successfully matched with the image frame, the first location and the location in the image database are considered to meet the preset location matching condition. The above is a place matching process, and the first place may be matched with all places in the image database, respectively.

When it is determined that a matching result satisfying a preset location matching condition cannot be obtained, the correspondence relationship between the first location, the first image frame and the feature information may be stored in the image database for use in subsequent matching.

S340: a location is determined from the third location that points to the same location as the first location.

The specific embodiment of this step may refer to the description of step S140, which is not repeated here.

In summary, in this embodiment, when each location in the image database corresponds to one image frame, each first image frame corresponding to the first location may be respectively matched with an image frame corresponding to each location in the image database, and when at least one first image frame is successfully matched with an image frame in the image database, a second matching result satisfying a preset location matching condition is determined to be obtained. Therefore, the embodiment can give consideration to the situation of single-view image frames in the image database, and improves the accuracy of loop detection.

In another embodiment of the invention, the embodiment shown in fig. 4 can be obtained on the basis of the embodiment shown in fig. 1. The method of the embodiment shown in fig. 4 may include the following steps.

S410: first image frames of a plurality of directions around a vehicle acquired at a first location by an image acquisition device are acquired. The specific explanation of this step can be referred to step S110, and will not be repeated here.

S420: and splicing the first image frames to obtain a first panoramic image frame at the first place.

When the first image frames are spliced, the first image frames can be spliced according to the preset position relation of the image acquisition devices and the preset position sequence, so that a first panoramic image frame containing the surrounding environment at the first place is obtained.

S430: and updating the pixel value of each pixel point according to the surrounding pixel points of each pixel point aiming at each pixel point in each first panoramic image frame, and taking all the updated pixel points in the first panoramic image frame as the characteristic information of the first panoramic image frame.

In this step, for each pixel point in each first panoramic image frame, the pixel value of the pixel point is updated according to the following formula:

P’＝(P-Pμ)/Pσ

Wherein P' is the updated pixel value of the pixel, P is the pixel value of the pixel, Pμ is the average value of the pixel values of the surrounding pixels of the pixel, and Pσ is the standard deviation of the pixel values of the surrounding pixels of the pixel.

The specific explanation of this step may also refer to the explanation in step S120, and will not be repeated here.

S440: and matching the first panoramic image frame with the panoramic image frames corresponding to a plurality of places in the image database according to the characteristic information of each panoramic image frame.

When the first panoramic image frame is successfully matched with the panoramic image frames in the image database, determining a third matching result meeting a preset place matching condition, wherein a fourth place indicated by the third matching result is: and the image database is used for matching the corresponding place of the panoramic image frame successfully.

Wherein the image database is used for storing characteristic information of panoramic image frames at a plurality of places.

In this embodiment, each location corresponds to a panoramic image frame. And when the matching is successful, determining to obtain a third matching result meeting the preset location matching condition.

S450: a location is determined from the fourth location that points to the same location as the first location. The specific embodiment of this step may refer to the explanation in step S140.

In summary, in this embodiment, each first image frame at the first location is stitched to obtain a first panoramic image frame, and when matching is performed, the first panoramic image frame is respectively matched with the panoramic image frames at each location in the image database. And each first image frame at the first location is spliced, so that the matching times between the panoramic image frame and each panoramic image frame in the image database can be reduced, the matching flow is shortened, and the matching efficiency is improved.

Fig. 5 is a schematic diagram of an apparatus for loop detection based on image frames according to an embodiment of the present invention. The apparatus corresponds to the embodiment of the method shown in fig. 1. The device is applied to electronic equipment. The device comprises:

an acquisition module 510 configured to acquire first image frames of a plurality of directions around the vehicle acquired by an image acquisition device at a first location;

A first updating module 520, configured to update, for each pixel point in each first image frame, a pixel value of the pixel point according to surrounding pixel points of the pixel point, and take all updated pixel points in the first image frame as feature information of the first image frame;

A first matching module 530 configured to match each of the first image frames with a plurality of image frames corresponding to a plurality of locations in the image database, respectively, according to the feature information of each of the image frames; when at least a preset number of successfully matched image frames exist in a plurality of image frames corresponding to the second location in the image database, determining to obtain a first matching result meeting a preset location matching condition; the location indicated by the first matching result is a second location; wherein the image database is used for storing characteristic information of image frames at a plurality of first-type places, and the image frames at each first-type place comprise image frames in a plurality of directions around a vehicle at the place;

A first determination module 540 is configured to determine a location from the second locations that is pointed to the same location as the first location.

In another embodiment of the present invention, based on the embodiment shown in fig. 5, when the second location is plural, the first determining module 540 is specifically configured to:

Judging whether target image frames which are continuous with image frames corresponding to a fifth location exist in the image frames corresponding to a plurality of second locations in the image database;

In another embodiment of the present invention, based on the embodiment shown in fig. 5, the first matching module 530 is specifically configured to:

calculating, for each first image frame and each image frame in an image database, an absolute value of a difference between a pixel value of each pixel point in the first image frame and a pixel value of a pixel point at the same position in the image frame, taking a sum of absolute values of differences between pixel values of corresponding two pixel points between the first image frame and the image frame as a feature information similarity between the first image frame and the image frame; and when the similarity of the characteristic information is smaller than a preset similarity threshold, determining that the first image frame is successfully matched with the image frame.

In another embodiment of the present invention, based on the embodiment shown in fig. 5, the apparatus further comprises:

A storage module (not shown in the figure) configured to store, when it is determined that a matching result satisfying the preset location matching condition cannot be obtained, feature information of each first image frame at the first location to the image database.

In another embodiment of the present invention, based on the embodiment shown in fig. 5, the first updating module 520 is specifically configured to:

P’＝(P-Pμ)/Pσ

In another embodiment of the present invention, based on the embodiment shown in fig. 5, the acquisition module 510 is specifically configured to:

In another embodiment of the invention, the embodiment shown in fig. 6 can be obtained based on the embodiment shown in fig. 5. In this embodiment, the image database is further configured to store feature information of image frames at a plurality of second-class places, each of the image frames at the second-class places including image frames of one direction around the vehicle at the place. This embodiment corresponds to the embodiment of the method shown in fig. 3. The device comprises: an acquisition module 610, a first update module 620, a second matching module 630, and a second determination module 640. The acquiring module 610 and the first updating module 620 are the same as the acquiring module 510 and the first updating module 520 in the embodiment shown in fig. 5, and detailed descriptions thereof are omitted.

A second matching module 630 configured to match each first image frame with one image frame at each location in the image database according to the feature information of each image frame after obtaining the feature information of the first image frame; when at least one first image frame exists in each first image frame and is successfully matched with the image frames in the image database, determining to obtain a second matching result meeting a preset place matching condition; the third location indicated by the second matching result is: the image database is matched with the place corresponding to the successfully matched image frame;

The second determination module 640 is configured to determine a location from the third location that points to the same location as the first location.

In a further embodiment of the invention, the embodiment of the device shown in fig. 7 can be obtained on the basis of the embodiment shown in fig. 5, which corresponds to the embodiment of the method shown in fig. 4. The apparatus further comprises: an acquisition module 710, a stitching module 720, a second update module 730, a third matching module 740, and a third determination module 750. The acquiring module 710 is the same as the acquiring module 510 in the embodiment shown in fig. 5, and detailed descriptions thereof are omitted here.

A stitching module 720, configured to stitch each first image frame after acquiring each first image frame acquired by the image acquisition device at the first location, so as to obtain a first panoramic image frame at the first location;

A second updating module 730, configured to update, for each pixel point in each first panoramic image frame, a pixel value of the pixel point according to surrounding pixel points of the pixel point, and take all updated pixel points in the first panoramic image frame as feature information of the first panoramic image frame;

A third matching module 740 configured to match the first panoramic image frame with panoramic image frames corresponding to a plurality of places in the image database according to the feature information of each panoramic image frame; when the first panoramic image frame is successfully matched with the panoramic image frames in the image database, determining a third matching result meeting a preset place matching condition, wherein a fourth place indicated by the third matching result is: the image database is used for matching the corresponding place of the panoramic image frame successfully; the image database is used for storing characteristic information of panoramic image frames at a plurality of places;

the third determination module 750 is configured to determine a location from the fourth location that points to the same location as the first location.

The device embodiment corresponds to the method embodiment, and has the same technical effects as the method embodiment, and the specific description refers to the method embodiment. The apparatus embodiments are based on the method embodiments, and specific descriptions may be referred to in the method embodiment section, which is not repeated herein.

Fig. 8 is a schematic structural diagram of a vehicle-mounted terminal according to an embodiment of the present invention. The vehicle-mounted terminal comprises: a processor 810 and an image acquisition device 820; the processor 810 includes: the first matching module 13 is configured to determine a first determination result of the first updating module 12, the first obtaining module 11, and the first determining module 14.

An acquisition module 11 for acquiring first image frames of a plurality of directions around the vehicle acquired at a first location by the image acquisition device 820;

A first updating module 12, configured to update, for each pixel point in each first image frame, a pixel value of the pixel point according to surrounding pixel points of the pixel point, and take all updated pixel points in the first image frame as feature information of the first image frame;

a first matching module 13, configured to match each first image frame with a plurality of image frames corresponding to a plurality of places in the image database according to the feature information of each image frame; when at least a preset number of successfully matched image frames exist in a plurality of image frames corresponding to the second location in the image database, determining to obtain a first matching result meeting preset location matching conditions; the location indicated by the first matching result is the second location; wherein the image database is used for storing characteristic information of image frames at a plurality of first-type places, and the image frames at each first-type place comprise image frames in a plurality of directions around a vehicle at the place;

A first determining module 14 for determining a location from the second location that is pointed to the same location as the first location.

In another embodiment of the present invention, based on the embodiment shown in fig. 8, the image database is further configured to store feature information of image frames at a plurality of second-class sites, each of the image frames at the second-class sites including image frames of one direction around the vehicle at the site; the processor 810 further includes:

A second matching module (not shown in the figure) for matching each first image frame with one image frame at each place in the image database according to the feature information of each image frame after the feature information of the first image frame is obtained; when at least one first image frame exists in each first image frame and is successfully matched with the image frames in the image database, determining to obtain a second matching result meeting a preset place matching condition; wherein, the third place indicated by the second matching result is: the image database is used for matching the corresponding place of the successfully matched image frame;

A second determining module (not shown in the figure) for determining a location from the third location that points to the same location as the first location.

In another embodiment of the present invention, based on the embodiment shown in fig. 8, the processor 810 further includes:

A stitching module (not shown in the figure) for stitching each first image frame after acquiring each first image frame acquired by the image acquisition device at the first location, so as to obtain a first panoramic image frame at the first location;

A second updating module (not shown in the figure) configured to update, for each pixel point in each first panoramic image frame, a pixel value of the pixel point according to surrounding pixel points of the pixel point, and taking all updated pixel points in the first panoramic image frame as feature information of the first panoramic image frame;

A third matching module (not shown in the figure) for matching the first panoramic image frame with the panoramic image frames corresponding to a plurality of places in the image database according to the feature information of each panoramic image frame; when the first panoramic image frame is successfully matched with the panoramic image frames in the image database, determining a third matching result meeting a preset place matching condition, wherein a fourth place indicated by the third matching result is: the image database is used for matching the corresponding place of the panoramic image frame successfully; the image database is used for storing characteristic information of panoramic image frames at a plurality of places;

a third determining module (not shown in the figure) is configured to determine a location pointing to the same location as the first location from the fourth location.

In another embodiment of the present invention, based on the embodiment shown in fig. 8, when the second location is plural, the first determining module 14 is specifically configured to:

In another embodiment of the present invention, based on the embodiment shown in fig. 8, the first matching module 13 is specifically configured to:

A storage module (not shown in the figure) for storing the feature information of each first image frame at the first location to the image database when it is determined that the matching result satisfying the preset location matching condition cannot be obtained.

In another embodiment of the present invention, based on the embodiment shown in fig. 8, the first updating module 12 is specifically configured to:

P’＝(P-Pμ)/Pσ

In another embodiment of the present invention, based on the embodiment shown in fig. 8, the obtaining module 11 is specifically configured to:

Initial image frames of a plurality of directions around the vehicle acquired at the first location by the image acquisition device 820 are acquired, and downsampling processing is performed on each initial image frame to obtain each first image frame.

The terminal embodiment and the method embodiment shown in fig. 1 are embodiments based on the same inventive concept, and the relevant points can be referred to each other. The terminal embodiment corresponds to the method embodiment, and has the same technical effects as the method embodiment, and the specific description refers to the method embodiment.

Those of ordinary skill in the art will appreciate that: the drawing is a schematic diagram of one embodiment and the modules or flows in the drawing are not necessarily required to practice the invention.

Those of ordinary skill in the art will appreciate that: the modules in the apparatus of the embodiments may be distributed in the apparatus of the embodiments according to the description of the embodiments, or may be located in one or more apparatuses different from the present embodiments with corresponding changes. The modules of the above embodiments may be combined into one module, or may be further split into a plurality of sub-modules.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A loop detection method based on image frames, comprising:

Acquiring first image frames of a plurality of directions around a vehicle acquired at a first place by an image acquisition device;

determining a location from the second location that points to the same location as the first location;

When the second location is plural, the step of determining a location from the second location that points to the same location as the first location includes:

2. The method of claim 1, wherein the image database is further configured to store characteristic information of image frames at a plurality of locations of a second type, each image frame at a location of a second type comprising an image frame of a direction surrounding a vehicle at the location; after obtaining the feature information of the first image frame, the method further includes:

3. The method of any of claims 1-2, wherein after acquiring each first image frame acquired by the image acquisition device at the first location, the method further comprises:

4. The method of claim 1, wherein the step of matching the respective first image frame with the plurality of image frames corresponding to each location in the image database based on the characteristic information of each image frame, comprises:

5. The method of claim 1, wherein when it is determined that a matching result satisfying the preset location matching condition cannot be obtained, the method further comprises:

and storing the characteristic information of each first image frame at the first location to the image database.

6. The method of claim 1, wherein the step of updating the pixel value of the pixel according to surrounding pixels of the pixel for each pixel in each first image frame comprises:

P’＝(P-Pμ)/Pσ

7. The method of any one of claims 1-6, wherein the step of acquiring a first image frame of a plurality of directions around the vehicle acquired by the image acquisition device at a first location comprises:

Acquiring initial image frames of a plurality of directions around the vehicle acquired by an image acquisition device at a first place;

And carrying out downsampling processing on each initial image frame to obtain each first image frame.

8. A loop detection apparatus based on image frames, comprising:

An acquisition module configured to acquire first image frames of a plurality of directions around a vehicle acquired at a first location by an image acquisition apparatus;

a first determination module configured to determine a location from the second locations that is pointed to the same location as the first location;

when the second location is plural, the determining a location from the second location that points to the same location as the first location includes:

9. A vehicle-mounted terminal, characterized by comprising: a processor and an image acquisition device; the processor includes: the device comprises an acquisition module, a first updating module, a first matching module and a first determining module;

The acquisition module is used for acquiring first image frames of a plurality of directions around the vehicle acquired by the image acquisition equipment at a first place;

The first determining module is used for determining a place which points to the same place as the first place from the second place;