CN111046743A

CN111046743A - Obstacle information labeling method and device, electronic equipment and storage medium

Info

Publication number: CN111046743A
Application number: CN201911151569.7A
Authority: CN
Inventors: 程邦胜; 方晓波; 张辉
Original assignee: Newpoint Enterprise Management Group Co Ltd
Current assignee: Newpoint Enterprise Management Group Co Ltd
Priority date: 2019-11-21
Filing date: 2019-11-21
Publication date: 2020-04-21
Anticipated expiration: 2039-11-21
Also published as: CN111046743B

Abstract

The application provides a method and a device for marking barrier information, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a video image acquired by a camera, a projection transformation matrix of the camera and a road coordinate system corresponding to a scene shot by the camera; determining a target obstacle in the video image; under the condition that the target barrier does not have initialized three-dimensional labeling information, sequentially acquiring 4 anchor points meeting geometric constraint conditions according to a three-dimensional coordinate system where the camera is located and a road surface coordinate system corresponding to a scene shot by the camera; initializing the attitude angle of the three-dimensional rectangular marking frame as an initial attitude angle; obtaining initial three-dimensional coordinates corresponding to the 4 anchor points; and determining the three-dimensional labeling information of the target obstacle. The method has the advantages of low cost, high information accuracy and strong obstacle distinguishing capability, and can meet the requirement of three-dimensional accurate positioning of the target obstacle in the road in various road-related applications.

Description

Obstacle information labeling method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a method and an apparatus for labeling obstacle information, an electronic device, and a storage medium.

Background

In technologies such as digital rail, automatic driving, vehicle-road coordination, robot navigation and the like, a camera mounted on a vehicle or a road side unit senses road condition information of a road area monitored by the camera, obtains a video image about the road condition, and then obtains related information of obstacles on a monitored road through analysis of the video image.

In the related art, a method for labeling a target obstacle in a video image with information of a two-dimensional area is provided, but the method can only determine two-dimensional information of the obstacle in a two-dimensional image coordinate system, so that three-dimensional information of the target obstacle cannot be obtained, and the requirement for accurately positioning the obstacle in a three-dimensional space, which is required in road condition analysis in automatic driving, vehicle-road coordination and auxiliary driving, is difficult to meet.

In the related art, a way of determining three-dimensional information of an obstacle depending on a laser radar sensor is also provided, but the way is high in cost, low in resolution of the laser radar, and difficult to accurately position the obstacle in the road.

Disclosure of Invention

The application provides a method and a device for marking barrier information, electronic equipment and a storage medium, so as to solve the technical problems.

The first aspect of the present application provides an obstacle information labeling method, where the method includes:

acquiring a video image acquired by a camera, a projection transformation matrix of the camera and a road coordinate system corresponding to a scene shot by the camera;

determining a target obstacle in the video image;

under the condition that the target barrier does not have initialized three-dimensional labeling information, sequentially acquiring 4 anchor points meeting geometric constraint conditions according to a three-dimensional coordinate system where the camera is located and a road surface coordinate system corresponding to a scene shot by the camera; the geometrical constraint conditions are as follows: three line segments corresponding to 3 line segments formed by the 4 anchor points according to the acquisition sequence in a three-dimensional space are mutually vertical, so that a three-dimensional rectangular space area occupied by the target barrier is consistent with a three-dimensional rectangular space area occupied by a three-dimensional rectangular marking frame formed by the 4 anchor points;

initializing the attitude angle of the three-dimensional rectangular marking frame as an initial attitude angle; obtaining initial three-dimensional coordinates corresponding to the 4 anchor points according to the initial two-dimensional coordinates of the 4 anchor points in the video image, a projective transformation matrix of the camera, a road coordinate system corresponding to a scene shot by the camera and an initial attitude angle of the three-dimensional rectangular marking frame; wherein, the attitude angle of the three-dimensional rectangular marking frame is as follows: an included angle between the forward direction of the three-dimensional rectangular marking frame and the forward direction of the road surface coordinate system;

and determining the three-dimensional labeling information of the target obstacle according to the initial three-dimensional coordinates corresponding to the 4 anchor points and the attitude angle of the three-dimensional rectangular labeling frame, and labeling the three-dimensional labeling information of the target obstacle in the video image.

The second aspect of the present application provides an obstacle information labeling apparatus, the apparatus including:

the first acquisition module is used for acquiring a video image acquired by a camera, a projection transformation matrix of the camera and a road coordinate system corresponding to a scene shot by the camera;

a first determination module for determining a target obstacle in the video image;

the anchor point acquisition module is used for sequentially acquiring 4 anchor points meeting geometric constraint conditions according to a three-dimensional coordinate system where the camera is located and a road surface coordinate system corresponding to a scene shot by the camera under the condition that the target barrier does not have initialized three-dimensional labeling information; the geometrical constraint conditions are as follows: three line segments corresponding to 3 line segments formed by the 4 anchor points according to the acquisition sequence in a three-dimensional space are mutually vertical, so that a three-dimensional rectangular space area occupied by the target barrier is consistent with a three-dimensional rectangular space area occupied by a three-dimensional rectangular marking frame formed by the 4 anchor points;

the second acquisition module is used for initializing the attitude angle of the three-dimensional rectangular marking frame as an initial attitude angle; obtaining initial three-dimensional coordinates corresponding to the 4 anchor points according to the initial two-dimensional coordinates of the 4 anchor points in the video image, the projective transformation matrix of the camera, the road coordinate system information of the scene shot by the camera and the initial attitude angle of the three-dimensional rectangular marking frame; wherein, the attitude angle of the three-dimensional rectangular marking frame is as follows: an included angle between the forward direction of the three-dimensional rectangular marking frame and the forward direction of the road surface coordinate system;

and the first labeling module is used for determining the three-dimensional labeling information of the target obstacle according to the initial three-dimensional coordinates, the three-dimensional rectangular labeling frame and the attitude angle of the three-dimensional rectangular labeling frame which are respectively corresponding to the 4 anchor points, and labeling the three-dimensional labeling information of the target obstacle in the video image.

A third aspect of the present application provides an electronic device comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute to implement an obstacle information labeling method.

A fourth aspect of the present application provides a non-transitory computer-readable storage medium, wherein instructions of the storage medium, when executed by a processor of a terminal, enable the terminal to perform a method for implementing an obstacle information annotation method.

Compared with the prior art, the method has the following advantages:

the method comprises the steps of determining a target obstacle of a video image according to the video image acquired by a camera and a projection transformation matrix, selecting a corresponding anchor point of the target obstacle from the video image, obtaining a three-dimensional rectangular marking frame of the target obstacle, and obtaining the three-dimensional marking information of the target obstacle by obtaining the position, size and posture information of the three-dimensional rectangular marking frame. The method and the device rely on the video image to obtain the three-dimensional labeling information of the target barrier, have low cost, high information accuracy and strong barrier distinguishing capability, and can meet the requirements of three-dimensional accurate positioning of the target barrier in the road in the applications of automatic driving, vehicle-road cooperation, auxiliary driving and the like.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.

Fig. 1 is a flowchart illustrating steps of a method for annotating obstacle information according to an embodiment of the present invention;

fig. 2 is a flowchart illustrating steps of a method for labeling obstacle information according to an embodiment of the present invention, in which an anchor point is determined first and then an attitude angle is adjusted;

FIG. 3 is a schematic diagram of a road coordinate system provided by an embodiment of the present invention;

FIG. 4 is a three-dimensional rectangular labeling box under a two-dimensional display coordinate system according to an embodiment of the present invention;

FIG. 5 is a three-dimensional rectangular marking frame in a road coordinate system according to an embodiment of the present invention;

FIG. 6 is a flow chart of the present invention for the specific 4 anchor points p1, p2, p3 and p4 shown in FIGS. 4 and 5;

FIG. 7 is a three-dimensional rectangular labeling box in a two-dimensional display coordinate system according to another embodiment of the present invention;

FIG. 8 is a three-dimensional rectangular marking frame in a road coordinate system according to another embodiment of the present invention;

FIG. 9 is a flow chart of the present invention for determining three-dimensional annotation information with initialized three-dimensional annotation information;

fig. 10 is a schematic structural diagram of an obstacle information labeling apparatus according to another embodiment of the present invention;

fig. 11 is a schematic structural diagram of an electronic device according to another embodiment of the present invention;

fig. 12 is a schematic structural diagram of an obstacle information labeling system according to another embodiment of the present invention.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present application more comprehensible, the present application is described in further detail with reference to the accompanying drawings and the detailed description.

In technologies such as automatic driving, vehicle-road cooperation, auxiliary driving and the like, an algorithm model needs to be trained to detect and locate obstacles in a video. The training of the models requires labeling of video data acquired by the camera, and marking the position and size of an obstacle in the video image in the image. Typically, the video image is displayed in a display window, and then a two-dimensional area of the video image in which the target obstacle is located is marked in the display window. And then storing the labeled two-dimensional labeling information in a storage device.

In the related art, a target obstacle in a video image can be labeled by using two-dimensional information. The method for labeling the target obstacle by using the two-dimensional information specifically comprises the following steps: the camera collects video images of the road area, and a two-dimensional area occupied by the target barrier is marked in the video images. However, the two-dimensional labeling information labeled by this method cannot reflect the information of the target obstacle in the video image in the three-dimensional space. If the video image marked with the two-dimensional marking information is adopted to train the detection model or the positioning model or the related video analysis model, the trained model can only position the two-dimensional information of the target obstacle under the two-dimensional image coordinate system, and the three-dimensional marking information of the target obstacle in the three-dimensional space can not be obtained, so that the model is difficult to carry out three-dimensional positioning on the target obstacle in the camera, and is difficult to meet the requirements of carrying out accurate positioning on the target obstacle in the three-dimensional space, which are required in road condition analysis in automatic driving, vehicle-road cooperation and auxiliary driving.

Therefore, the above-mentioned related technology of "labeling the target obstacle in the video image by using the two-dimensional information" can only label the two-dimensional area information of the target obstacle in the video image in the two-dimensional image coordinate system, and cannot label the three-dimensional information of the target obstacle in the video image in the three-dimensional space, therefore, the obtained annotation data does not contain the three-dimensional information of the target obstacle in the video image of the monitored road area, and cannot be used for training a three-dimensional detection or positioning model or a related video analysis model of the target obstacle, and the three-dimensional space information of the target obstacle cannot be obtained through the analysis of the video image, so that the obstacle in the video image cannot be accurately positioned in the three-dimensional space, and the requirement of three-dimensional accurate positioning of the target obstacle in the road in the applications of automatic driving, vehicle-road cooperation, auxiliary driving and the like cannot be met.

In the related art, another way of obtaining three-dimensional information of an obstacle depending on three-dimensional point cloud information sensed by a laser radar sensor is provided. Namely, the point cloud information based on the laser radar contains three-dimensional information of sampling points of the monitored road area, and the target obstacle in the three-dimensional point cloud can be labeled in a target area of the three-dimensional space area. Because the three-dimensional point cloud contains the three-dimensional coordinate information of the road area relative to the laser radar coordinate system, the three-dimensional information of the target obstacle can be labeled, including the position information of the target obstacle, the occupied three-dimensional space area information and the three-dimensional attitude information.

However, although the above-mentioned method of obtaining three-dimensional information of an obstacle by relying on three-dimensional point cloud information sensed by a lidar sensor can mark three-dimensional information on a target obstacle on a road by point cloud data of a monitored road sensed by the lidar sensor, the lidar sensor is expensive, and has a low resolution compared with a video image resolution, so that it is difficult to accurately locate the target obstacle on the road, and the location of a three-dimensional space area occupied by the target obstacle in the three-dimensional space is also lower than that of the video image. In addition, the laser radar cannot sense color information on a road area, and the accuracy of classifying the target obstacle is lower than that of classifying the target obstacle based on the video image. For example, if there are people, a box body and a vehicle on the road surface, wherein the difference between people and the box body is not large, if the target obstacle is located by relying on the laser radar, the people and the box body are easily confused due to the fact that color information cannot be sensed, and the accuracy of classification of the target obstacle is further reduced.

The three-dimensional labeling information of the obstacle comprises position information of the obstacle, three-dimensional space information occupied by the obstacle and attitude information of the obstacle. The three-dimensional marking information is important for positioning and analyzing the obstacle in the three-dimensional space, and only by accurately positioning the position of the obstacle in the three-dimensional space and accurately analyzing the state of the obstacle, the automatic driving vehicle can make a correct driving decision according to the information, avoid the obstacle near the vehicle and ensure that the vehicle safely and smoothly runs.

To analyze the three-dimensional spatial information of the obstacle in the monocular video image of the monitored road area, the monocular video data marked with the three-dimensional information of the obstacle needs to be used for training a video analysis model. Therefore, there is a need for a related technology and system for labeling three-dimensional information of an obstacle in a monocular video, and accurately reflecting position information of the obstacle in a three-dimensional space, occupied three-dimensional space region information, and posture information in the three-dimensional space in a monocular video image.

In order to solve the above technical problems, the present application provides an obstacle information labeling method, which determines a target obstacle of a video image according to the video image and a projection transformation matrix acquired by a camera, selects a corresponding anchor point of the target obstacle from the video image, obtains a three-dimensional rectangular labeling frame of the target obstacle, and obtains three-dimensional labeling information of the target obstacle by obtaining position, size, and posture information of the three-dimensional rectangular labeling frame.

Compared with the mode of acquiring the three-dimensional information by the laser radar in the related technology, the method and the device rely on the video image to acquire the three-dimensional labeling information of the target obstacle, can acquire the three-dimensional labeling information of the target obstacle only by relying on the video image, and can easily acquire the video image with higher resolution, so that the cost of the labeling method provided by the application is low; the three-dimensional labeling information of the target obstacle is obtained based on the video image, and the resolution of the video image can be very high, so that the three-dimensional information of the target obstacle can be more accurately obtained based on the high-resolution video image; and the video image can identify and record the color of the target barrier, and the barrier distinguishing capability can be effectively improved. Compared with the mode of marking the obstacle through the two-dimensional information in the related technology, the method and the device can obtain accurate three-dimensional marking information, and therefore the requirement that three-dimensional accurate positioning is needed to be carried out on the target obstacle in the road in the applications such as automatic driving, vehicle-road cooperation and auxiliary driving can be met.

The method for labeling obstacle information provided by the application is as shown in fig. 1 and fig. 2, and specifically includes the following steps:

and step S11, acquiring a video image acquired by a camera, a projection transformation matrix of the camera and a road surface coordinate system corresponding to a scene shot by the camera.

The video collected by the camera is formed by one frame of video image, and the obstacle of each frame of video image in the video is labeled, so that the labeling of the obstacle appearing in the whole video is completed.

According to the position and the posture of the camera relative to the three-dimensional global coordinate system and the internal parameter values of the camera, the three-dimensional coordinate of a point in a three-dimensional space under the three-dimensional global coordinate system and the coordinate transformation relation between the two-dimensional coordinates of the corresponding two-dimensional points in the two-dimensional plane of the imaging coordinate system on the imaging surface of the camera can be obtained for the given camera, and the coordinate transformation relation can be expressed as a projection transformation matrix. This projective transformation matrix may be represented as a 3 × 4 homogeneous matrix P, typically in the form of K [ R | t ], K being a 3 × 3 internal reference matrix, R being a 3 × 3 rotation matrix, and t being a 3 × 1 three-dimensional vector. The projection transformation matrix transforms the homogeneous coordinates of the 3-dimensional points into the homogeneous coordinates of the 2-dimensional points, and the process of projection transformation from three-dimensional to two-dimensional is completed.

And determining a three-dimensional coordinate system (namely a three-dimensional global coordinate system which refers to a world coordinate system) where the camera is located and a road surface coordinate system corresponding to the camera.

The road surface coordinate system corresponding to the camera is the road surface coordinate system of the road where the camera is located. The road surface coordinate system is obtained as follows:

as shown in fig. 3, the monitored road surface area is approximately a plane. With respect to the three-dimensional global coordinate system, the road pavement plane is described by a normal vector v _ n and a road forward vector v _ f of the road pavement, and a three-dimensional coordinate system of an anchor point a on the road pavement. The anchor point a may be any point of the road surface.

The left vector of the road surface can be calculated according to the normal vector v _ n of the road surface and the front direction vector v _ f of the road surface

Wherein

Is the cross product of the vectors. Wherein v _ n, v _ f and v _ l constitute three coordinate axes of a road pavement coordinate system.

Step S12, determining a target obstacle in the video image.

All obstacles appearing in one frame of video image can become target obstacles. In step S12, all obstacles for which three-dimensional labeling information needs to be determined may be determined as target obstacles, and then three-dimensional labeling information is determined for all target obstacles one by one; or only one target obstacle can be determined, and after the three-dimensional marking information of the target obstacle is determined, the next target obstacle needing to be marked with the three-dimensional marking information is determined, and the three-dimensional marking information is determined for the next target obstacle.

Step S13, under the condition that the target obstacle does not have initialized three-dimensional labeling information, sequentially acquiring 4 anchor points meeting geometric constraint conditions according to a three-dimensional coordinate system where the camera is located and a road surface coordinate system corresponding to a scene shot by the camera; the geometrical constraint conditions are as follows: and three line segments corresponding to the 3 line segments formed by the 4 anchor points according to the acquisition sequence in the three-dimensional space are mutually vertical, so that the three-dimensional rectangular space area occupied by the target barrier is consistent with the three-dimensional rectangular space area occupied by the three-dimensional rectangular labeling frame formed by the 4 anchor points.

After the target obstacle is determined, it is first determined whether the target obstacle has initialized three-dimensional labeling information, which is pre-existing. The initialized three-dimensional annotation information can be obtained by the method in the related art.

In the automatic driving, the vehicle-road cooperation, and the driving assistance, since the target obstacle is on the road surface on which the vehicle is traveling, there is a possibility that the vehicle may be caused to travel obstacle, it is necessary to determine information such as the position of the target obstacle in the road, the spatial region, and the like. Although the position and the space area occupied by the obstacle on the road are different according to the obstacle, the influence of the obstacle on the running vehicle can be determined by only obtaining the longest length of the obstacle in the horizontal direction, the longest length of the obstacle in the vertical direction and the occupied position in the three-dimensional global coordinate system. Therefore, the target obstacle is abstracted into the three-dimensional rectangular marking frame, and the three-dimensional information of the target obstacle is obtained by obtaining the three-dimensional information of the three-dimensional rectangular marking frame.

In the application, the three-dimensional rectangular marking frame is obtained as follows:

as shown in fig. 4, the three-dimensional rectangular label box is a two-dimensional display coordinate system (where the two-dimensional display coordinate system refers to two-dimensional coordinates on a screen of a display window presenting a video image, O is an origin of the two-dimensional display coordinate system, U is a horizontal axis of the two-dimensional display coordinate system, and V is a vertical axis of the two-dimensional display coordinate system). The method comprises the steps of sequentially selecting 4 anchor points P1, P2, P3 and P4 (namely the 4 anchor points are two-dimensional points in a two-dimensional coordinate system selected from a video image) in the vicinity of a target obstacle presented in the video image or on the edge of the target obstacle, wherein the 4 anchor points in the two-dimensional display coordinate system respectively correspond to one anchor point in a three-dimensional space, and the anchor points are respectively marked as P1, P2, P3 and P4. Anchor points P1, P2, P3 and P4 in the three-dimensional space need to meet geometric constraint conditions, wherein the geometric constraint conditions are that line segments P1-P2, P2-P3 and P3-P4 formed by 4 anchor points obtained according to the acquisition sequence are mutually vertical; therefore, 4 anchor points can be constrained to form a cuboid, and the cuboid is the three-dimensional rectangular labeling frame. As shown in fig. 5, the three-dimensional rectangular marking frame in the road surface coordinate system is shown.

As shown in fig. 6, fig. 6 is a flowchart of determining three-dimensional annotation information in conjunction with specific 4 anchor points p1, p2, p3 and p 4. The method comprises the following specific steps: inputting a video image, a camera projection transformation matrix and road pavement coordinate system information, and displaying the input video image in a display window; determining a target obstacle in the video image, sequentially selecting anchor points P1, P2, P3 and P4 in the display window, and simultaneously calculating three-dimensional coordinates of corresponding three-dimensional points P1, P2, P3 and P4; adjusting the positions of anchor points p1, p2, p3 and p4 in a two-dimensional coordinate system as required, and updating the three-dimensional coordinates of the three-dimensional points corresponding to the anchor points; adjusting the attitude angle according to the requirement to obtain the three-dimensional labeling information of the target barrier; determining the three-dimensional labeling information of the next target obstacle according to the method; and then transmitting or storing the obtained three-dimensional labeling information.

The positions of the 4 anchor points may be as shown in fig. 7, and fig. 8 may be a three-dimensional rectangular marking frame in the road surface coordinate system determined by the anchor point positions shown in fig. 7, and the three-dimensional marking information may be determined according to the flowchart for determining the three-dimensional marking information shown in fig. 6.

Because the target barrier is positioned in the three-dimensional rectangular marking frame, the vehicle can avoid the target barrier as long as the vehicle avoids the area of the three-dimensional rectangular marking frame in the specific automatic driving process.

In order to simplify the three-dimensional rectangular marking frame, all obstacles can be set to be on the road surface; the bottom surface of the three-dimensional rectangular marking frame can be overlapped with the road surface. Setting the forward direction of the three-dimensional rectangular marking frame to be consistent with the forward direction of the road, namely assuming that the attitude angle of the three-dimensional rectangular marking frame is 0, and subsequently adjusting the three-dimensional rectangular marking frame to ensure that the attitude angle of the three-dimensional rectangular marking frame is the same as the attitude angle of the target obstacle.

Step S14, initializing the attitude angle of the three-dimensional rectangular marking frame as an initial attitude angle; obtaining initial three-dimensional coordinates corresponding to the 4 anchor points according to the initial two-dimensional coordinates of the 4 anchor points in the video image, a projective transformation matrix of the camera, a road coordinate system corresponding to a scene shot by the camera and an initial attitude angle of the three-dimensional rectangular marking frame; wherein, the attitude angle of the three-dimensional rectangular marking frame is as follows: and the included angle between the forward direction of the three-dimensional rectangular marking frame and the forward direction of the road surface coordinate system.

The initial three-dimensional coordinates of the 4 anchor points in the three-dimensional global coordinate system are obtained according to the initial two-dimensional coordinates, the projection transformation matrix, the road surface coordinate system and the initial attitude angle.

Step S15, determining three-dimensional labeling information of the target obstacle according to the initial three-dimensional coordinates, the three-dimensional rectangular labeling frame, and the attitude angle of the three-dimensional rectangular labeling frame corresponding to each of the 4 anchor points, and labeling the three-dimensional labeling information of the target obstacle in the video image.

The three-dimensional labeling information comprises position information of the target obstacle, three-dimensional space information occupied by the target obstacle and posture information of the target obstacle.

The position information of the target obstacle is represented by a three-dimensional coordinate of an anchor point of the three-dimensional rectangular marking frame, wherein the anchor point can be a vertex set by the bottom surface of the three-dimensional rectangular marking frame, and can also be a central point of a bottom surface rectangle of the three-dimensional rectangular marking frame.

The three-dimensional space information occupied by the target obstacle refers to the space volume occupied by the three-dimensional rectangular marking frame.

The attitude information of the target obstacle refers to an included angle between the forward direction of the target obstacle and a forward direction vector of a road surface.

For example, the target obstacle is a person, the front of the person faces the forward direction of the road surface, and the angle between the forward direction of the person and the forward direction vector of the road surface is 0 °. The front of the human body is back to the front of the road pavement, and the included angle between the front of the human body and the front vector of the road pavement is 180 degrees.

After the three-dimensional labeling information is obtained, the three-dimensional labeling information can be labeled in the video image; the information may also be stored to an information storage device and/or transmitted to a target device via an information transfer device. The target device is a device that requires three-dimensional labeling information, such as an autonomous automobile.

Wherein, step S15 specifically includes:

step S1501, determining the positions and the sizes of the three-dimensional rectangular marking frames where the 4 anchor points are located according to the initial three-dimensional coordinates corresponding to the 4 anchor points respectively.

In step S14, knowing the initial three-dimensional coordinates of the 4 anchor points, the position and size of the three-dimensional rectangular marking frame can be determined, wherein the position of the three-dimensional rectangular marking frame is represented by the three-dimensional coordinates of one anchor point of the three-dimensional rectangular marking frame, and the anchor point may be a vertex set by the bottom surface of the three-dimensional rectangular marking frame, or a center point of a rectangle on the bottom surface of the three-dimensional rectangular marking frame, etc. The size of the three-dimensional rectangular marking frame is the volume of a three-dimensional space occupied by the length, the width and the height of the three-dimensional rectangular marking frame. The length refers to the length of the three-dimensional rectangular marking frame in the v _ f direction, the width refers to the width of the three-dimensional rectangular marking frame in the v _ l direction, and the height refers to the height of the three-dimensional rectangular marking frame in the v _ n direction. The three-dimensional attitude information of the obstacle is an angle theta between the front direction of the obstacle and the front direction v _ f of the road.

The forward direction of the three-dimensional rectangular labeling frame is determined by the forward direction of the corresponding target obstacle.

Step S1502, under the condition that the initial attitude angle of the three-dimensional rectangular marking frame is the same as the attitude angle of the target obstacle, determining three-dimensional marking information of the target obstacle according to the initial three-dimensional coordinates, the position and the size of the three-dimensional rectangular marking frame and the initial attitude angle of the three-dimensional rectangular marking frame corresponding to the 4 anchor points; wherein the attitude angle of the target obstacle is: an angle between a forward direction of the target obstacle and a forward direction of the road coordinate system.

When 4 anchor points are selected to construct the three-dimensional rectangular marking frame, the attitude angle of the three-dimensional rectangular marking frame is possibly different from the attitude angle of the target obstacle, and under the condition that the attitude angles of the three-dimensional rectangular marking frame and the target obstacle are different, the three-dimensional marking information obtained according to the three-dimensional rectangular marking frame is different from the actual three-dimensional marking information of the target obstacle.

In order to avoid such a difference, it is necessary to determine whether the attitude angle of the three-dimensional rectangular labeling frame is the same as the attitude angle of the target obstacle. The attitude angle of the three-dimensional rectangular marking frame is reflected after the attitude angle of the target obstacle is estimated, and the attitude angle of the three-dimensional rectangular marking frame is confirmed by whether the three-dimensional rectangular space occupied by the three-dimensional rectangular marking frame is consistent with the three-dimensional rectangular space occupied by the obstacle or not. If the attitude angle of the three-dimensional rectangular marking frame enables the three-dimensional rectangular space occupied by the three-dimensional rectangular marking frame to be consistent with the three-dimensional rectangular space occupied by the obstacle, the attitude angle of the three-dimensional rectangular marking frame is considered to reflect the attitude angle of the obstacle.

Under the condition that the initial attitude angle of the three-dimensional rectangular marking frame is the same as the attitude angle of the target obstacle, that is, under the condition that the initial attitude angle of the three-dimensional rectangular marking frame can make the three-dimensional rectangular space area occupied by the three-dimensional rectangular marking frame consistent with the three-dimensional rectangular space area occupied by the target obstacle, so that the attitude angle of the target obstacle can be correctly reflected, meaning that the three-dimensional rectangular marking frame can correctly represent the three-dimensional information of the target obstacle, the three-dimensional marking information of the target obstacle can be determined according to the initial three-dimensional coordinates, the positions and the sizes of the three-dimensional rectangular marking frame and the attitude angle of the three-dimensional rectangular marking frame corresponding to the 4 anchor points.

Step S1503, under the condition that the initial attitude angle of the three-dimensional rectangular marking frame is different from the attitude angle of the target obstacle, adjusting the initial attitude angle of the three-dimensional rectangular marking frame to enable the adjusted attitude angle of the three-dimensional rectangular marking frame to be the same as the attitude angle of the target obstacle;

that is, in a case where the initial attitude angle of the initial three-dimensional rectangular marking frame does not make the three-dimensional rectangular space area occupied by the three-dimensional rectangular marking frame consistent with the three-dimensional rectangular space area occupied by the target obstacle, and thus the attitude angle of the target obstacle cannot be correctly reflected, the initial attitude angle of the three-dimensional rectangular marking frame is adjusted so that the adjusted attitude angle of the three-dimensional rectangular marking frame makes the three-dimensional rectangular space area occupied by the three-dimensional rectangular marking frame consistent with the three-dimensional rectangular space area occupied by the obstacle.

Determining the adjusted two-dimensional coordinates of the 4 anchor points according to the adjusted attitude angle of the three-dimensional rectangular marking frame;

determining the adjusted three-dimensional coordinates corresponding to the adjusted attitude angles of the 4 anchor points according to the adjusted two-dimensional coordinates of the 4 anchor points, the projective transformation matrix of the camera, the road coordinate system corresponding to the scene shot by the camera and the adjusted attitude angles of the three-dimensional rectangular marking frame;

and determining the three-dimensional labeling information of the target obstacle according to the adjusted three-dimensional coordinates, the position and the size of the three-dimensional rectangular labeling frame and the posture angle of the adjusted three-dimensional rectangular labeling frame which are respectively corresponding to the 4 anchor points, and labeling the three-dimensional labeling information of the target obstacle in the video image.

Under the condition that the attitude angle of the three-dimensional rectangular marking frame is different from the attitude angle of the target obstacle, namely under the condition that the attitude angle of the three-dimensional rectangular marking frame cannot make the three-dimensional rectangular space area occupied by the three-dimensional rectangular marking frame consistent with the three-dimensional rectangular space area occupied by the target obstacle, so that the attitude angle of the target obstacle cannot be correctly reflected, the three-dimensional rectangular marking frame cannot correctly represent the three-dimensional information of the target obstacle, and the attitude angle of the three-dimensional rectangular marking frame needs to be adjusted.

In the marking process, the three-dimensional rectangular marking frame can be rotated on the road surface by continuously adjusting the initial value of the attitude angle until reaching an angle, so that the angle of the three-dimensional rectangular marking frame is consistent with the angle of the target obstacle, and at the moment, the position and the size of the three-dimensional rectangular marking frame can be adjusted, so that the three-dimensional rectangular space area represented by the three-dimensional rectangular marking frame is consistent with the three-dimensional rectangular space area occupied by the target obstacle.

And finally, enabling the angle, which is most consistent between the three-dimensional rectangular space region represented by the three-dimensional rectangular marking frame and the three-dimensional space region occupied by the target obstacle, to be the attitude angle capable of correctly reflecting the target obstacle.

For example, by changing the position of p1 in the display window, the position of the three-dimensional rectangular frame in the three-dimensional space can be changed. Further, the height of the three-dimensional rectangular frame can be changed by changing the position of p2 with respect to p1 in the display window, the length of the three-dimensional rectangular frame can be changed by changing the position of p3 with respect to p2 in the display window, and the width of the three-dimensional rectangular frame can be changed by changing the position of p4 with respect to p3 in the display window.

In addition, the two-dimensional coordinates and the three-dimensional coordinates of the 4 anchor points are in one-to-one correspondence, so that the adjusted three-dimensional rectangular marking frame can determine the information such as the position and the size of the three-dimensional rectangular marking frame according to the relationship between the two-dimensional coordinates and the three-dimensional coordinates.

In addition, on the basis that the size of the three-dimensional rectangular marking frame is determined, the whole three-dimensional rectangular marking frame can be driven to rotate or move by moving the two-dimensional coordinates of the 1 anchor point of the three-dimensional rectangular marking frame, so that the attitude angle of the three-dimensional rectangular marking frame can be adjusted, and the attitude angle of the three-dimensional rectangular marking frame is consistent with the attitude angle of the target obstacle.

If the direction of the obstacle is not consistent with the direction of v _ f, the three-dimensional rectangular frame can be rotated counterclockwise around the v _ n axis by the angle theta by adjusting the angle theta of the attitude angle of the three-dimensional rectangular marking frame, so that the direction of the three-dimensional rectangular marking frame is consistent with the direction of the obstacle.

Under the condition that the attitude angle of the three-dimensional rectangular marking frame is the same as the attitude angle of the target obstacle, meaning that the three-dimensional rectangular marking frame can correctly represent the three-dimensional information of the target obstacle, the three-dimensional marking information of the target obstacle can be determined according to the adjusted three-dimensional coordinates, the adjusted position and size of the three-dimensional rectangular marking frame and the adjusted attitude angle of the three-dimensional rectangular marking frame corresponding to the 4 anchor points.

In another embodiment provided by the present application, after determining the target obstacle, it is first required to determine whether the target obstacle has initialized three-dimensional labeling information, where the initialized three-dimensional labeling information is pre-existing. The initialized three-dimensional labeling information can be obtained by the method in the related art, and can also be given by the following method provided in the application, specifically as follows:

and step S21, acquiring a video image acquired by a camera, a projection transformation matrix of the camera and a road surface coordinate system corresponding to a scene shot by the camera.

Step S21 is similar to step S11, please refer to step S11, and further description is omitted here.

Step S22, determining a target obstacle in the video image.

Step S22 is similar to step S12, please refer to step S12, and further description is omitted here.

Step S23, estimating the attitude angle of the target obstacle according to the orientation of the target obstacle to obtain the estimated attitude angle of the target obstacle; wherein the attitude angle of the target obstacle is: the angle between the forward direction of the target obstacle and the forward direction of the road coordinate system.

And (4) estimating the attitude angle of the target obstacle in the video image to obtain an estimated attitude angle which is not very accurate. For example, if the target obstacle is a person, the orientation of the face of the person is assumed to be forward, and if the face of the person is assumed to be forward of the road, the estimated attitude angle is 0. For another example, if the target obstacle is a car parked on the roadside and the direction of the vehicle head is parallel to the left-hand vector of the road surface, the angle between the forward direction of the car and the forward direction of the road is 90 °, the estimated attitude angle is 90 °. For another example, if the target obstacle is another automobile parked on the roadside and the direction of the head is inclined, the angle between the forward direction of the automobile and the forward direction of the road is 45 °, and the estimated attitude angle is 45 °.

Step S24, sequentially acquiring 4 initial anchor points meeting geometric constraint conditions according to the estimated attitude angle of the target obstacle, the three-dimensional coordinate system where the camera is located and the road surface coordinate system corresponding to the scene shot by the camera; the geometrical constraint conditions are as follows: and three line segments corresponding to the 3 line segments formed by the 4 initial anchor points according to the acquisition sequence in the three-dimensional space are mutually vertical, so that the three-dimensional rectangular space area occupied by the target barrier is consistent with the three-dimensional rectangular space area occupied by the initial three-dimensional rectangular labeling frame formed by the 4 initial anchor points.

And taking the estimated attitude angle of the target obstacle as a reference, sequentially obtaining 4 initial anchor points meeting geometric constraint conditions according to the three-dimensional coordinate system where the camera is located and the road surface coordinate system corresponding to the camera, constructing an initial three-dimensional rectangular frame by using the 4 initial anchor points, wherein the method for constructing the initial three-dimensional rectangular frame is the same as the method for obtaining the three-dimensional rectangular frame in the step S13, and details are not repeated here.

Step S25, obtaining initial three-dimensional coordinates corresponding to the 4 initial anchor points according to the initial two-dimensional coordinates of the 4 initial anchor points in the video image, the projective transformation matrix of the camera, the road coordinate system corresponding to the scene shot by the camera, and the estimated attitude angle of the target obstacle.

Step S25 is similar to step S14, please refer to step S14, and further description is omitted here.

Step S26, determining initialized three-dimensional labeling information of the target obstacle according to the initial three-dimensional coordinates, the position and the size of the initial three-dimensional rectangular labeling frame, and the estimated attitude angle, which correspond to the 4 initial anchor points, respectively.

Step S26 is similar to step S15, please refer to step S15, and further description is omitted here.

Step S27, labeling the initialized three-dimensional labeling information of the target obstacle in the video image.

When the target obstacle has initialized three-dimensional labeling information, it is necessary to determine whether the initialized three-dimensional labeling information can accurately reflect information of the target obstacle. If the three-dimensional rectangular marking frame determined by the three-dimensional marking information can completely cover the target obstacle in the three-dimensional rectangular marking frame, the initialized three-dimensional marking information can accurately reflect the information of the target obstacle, otherwise, the initialized three-dimensional marking information cannot be used.

Specifically, as shown in fig. 9, fig. 9 is a flowchart for determining three-dimensional labeling information in the case of having initialized three-dimensional labeling information, and in the case of having initialized three-dimensional labeling information in the target obstacle, the method specifically includes:

step S31, obtaining an initialized three-dimensional rectangular marking frame according to the initialized three-dimensional marking information;

step S32, labeling the initialized three-dimensional labeling information in the video image when the three-dimensional rectangular space area occupied by the target obstacle is consistent with the three-dimensional rectangular space area occupied by the initialized three-dimensional rectangular labeling frame.

Step S33, when the three-dimensional rectangular space area occupied by the target obstacle is not consistent with the three-dimensional rectangular space area occupied by the initialized three-dimensional rectangular labeling frame, adjusting the positions of at least 1 anchor point in the initialized three-dimensional rectangular labeling frame and/or the attitude angle of the three-dimensional rectangular labeling frame, so that the three-dimensional rectangular space area occupied by the target obstacle is consistent with the three-dimensional rectangular space area occupied by a new three-dimensional rectangular labeling frame formed by the adjusted new positions of 4 anchor points.

Namely, the positions of the 4 initial anchor points are adjusted to obtain the new positions of the 4 anchor points, and a new three-dimensional rectangular marking frame is determined according to the 4 new anchor points.

And adjusting the position of at least 1 anchor point in the initialized three-dimensional rectangular marking frame and/or the attitude angle of the three-dimensional rectangular marking frame, namely in a display window for displaying the video image, and changing the length, width, height and attitude angle of the three-dimensional rectangular marking frame by adjusting the position of the two-dimensional coordinate of the anchor point in the two-dimensional display coordinate system.

The conditions and principles for determining the 4 new anchor points in step S33 are the same as those in step S13, please refer to the related contents in step S13, and further description is omitted here.

Step S34, initializing the attitude angle of the three-dimensional rectangular marking frame as an initial attitude angle; and obtaining initial three-dimensional coordinates corresponding to the adjusted new positions of the 4 anchor points according to the initial two-dimensional coordinates of the adjusted new positions of the 4 anchor points in the video image, the projective transformation matrix of the camera, the road coordinate system corresponding to the scene shot by the camera and the initial attitude angle of the three-dimensional rectangular marking frame. Wherein the attitude angle of the new three-dimensional rectangular marking frame is as follows: the included angle between the forward direction of the new three-dimensional rectangular marking frame and the forward direction of the road surface coordinate system

Step S34 is similar to step S14, please refer to step S14, and further description is omitted here.

And step S35, determining the position and size of the new three-dimensional rectangular marking frame according to the initial three-dimensional coordinates corresponding to the adjusted new positions of the 4 anchor points.

Step S35 is similar to step S1501, please refer to step S1501, and further description is omitted here.

Step S36, under the condition that the attitude angle of the new three-dimensional rectangular marking frame is the same as the attitude angle of the target obstacle, determining new three-dimensional marking information of the target obstacle according to the initial three-dimensional coordinates corresponding to the adjusted new positions of the 4 anchor points, the position and the size of the new three-dimensional rectangular marking frame and the attitude angle of the new three-dimensional rectangular marking frame, and marking the new three-dimensional marking information in the video image; wherein the attitude angle of the target obstacle is: an angle between a forward direction of the target obstacle and a forward direction of the road coordinate system.

Step S36 is similar to step S1502 in principle, please refer to the relevant content of step S1502, and will not be described herein again.

Step S37, under the condition that the attitude angle of the new three-dimensional rectangular marking frame is not the same as the attitude angle of the target obstacle, adjusting the initial attitude angle of the new three-dimensional rectangular marking frame to ensure that the adjusted attitude angle of the new three-dimensional rectangular marking frame is the same as the attitude angle of the target obstacle;

determining adjusted three-dimensional coordinates corresponding to the adjusted 4 new anchor points according to the adjusted attitude angle of the new three-dimensional rectangular marking frame and the projection transformation matrix;

and determining the three-dimensional labeling information of the target obstacle according to the adjusted three-dimensional coordinates, the position and the size of the new three-dimensional rectangular labeling frame and the attitude angle of the adjusted new three-dimensional rectangular labeling frame which are respectively corresponding to the adjusted 4 anchor points, and labeling the new three-dimensional labeling information of the target obstacle in the video image.

Step S37 is similar to step S1503, please refer to the related contents of step S1503, and will not be described herein again.

Specifically, the adjustment process of the attitude angle θ of the three-dimensional rectangular marking frame and the adjustment process of the positions of the anchor points p1, p2, p3 and p4 in the display window may be continuously performed until the marked three-dimensional rectangular marking frame can accurately reflect the position of the marked obstacle in the three-dimensional space and the occupied space area in the three-dimensional space; and then, storing the position information, the length, the width, the height and the attitude angle information of the three-dimensional rectangular marking frame in the three-dimensional space as the three-dimensional marking information of the marked barrier.

The process is repeated for all the obstacles in the video image, and then the three-dimensional information of all the obstacles can be labeled. Once the labeling of the three-dimensional information of all the obstacles in the video image is finished, the labeling of the three-dimensional information of the video image is finished, and the corresponding three-dimensional labeling information is stored in the information storage device and/or is sent to the target device through the information transmission device.

Through the implementation mode of the embodiment, the three-dimensional information of the obstacle on the road surface can be accurately and effectively marked on the monocular video image, so that a monocular video data set with the three-dimensional information mark can be obtained, the model algorithm which can be used for detecting from the monocular video image and positioning the obstacle in a three-dimensional space is trained, the obstacle on the road surface can be detected and accurately positioned through the monocular camera in application scenes such as vehicle-road cooperation, automatic driving and robot obstacle avoidance, the cost of technologies such as vehicle-road cooperation, automatic driving and robot obstacle avoidance is greatly reduced, and the positioning accuracy of the obstacle is improved. The driving safety of the automatic driving vehicle is guaranteed.

The method and the device can adjust and modify the existing three-dimensional annotation information of the video image, and ensure the accuracy of the three-dimensional annotation information of the video image; the video image can be pre-marked with three-dimensional information by an automatic tool, and then is adjusted and modified, so that the efficiency of marking the three-dimensional information of the video image is improved, and the cost of marking the three-dimensional information of the video image is reduced.

The method and the device have the advantages that a three-dimensional rectangular frame is used for marking an obstacle in the video image, and the position of the obstacle in a three-dimensional space, the occupied three-dimensional space area and three-dimensional posture information are reflected. Wherein, the bottom surface of three-dimensional rectangle frame and road surface coincide.

Firstly, inputting a video image to be marked, a corresponding camera projection transformation matrix and road surface coordinate system information, displaying the video image to a display window, and if the initial three-dimensional marking information exists, displaying the initial three-dimensional marking information to the display window.

After a target obstacle to be marked in a video image is determined, selecting the position of a first anchor point in a display window, constraining the anchor point on a road surface, calculating the three-dimensional coordinate of a first three-dimensional point corresponding to the first anchor point in a three-dimensional space according to a camera projection transformation matrix and road surface coordinate system information, constraining the first three-dimensional point to be a vertex on a three-dimensional rectangular marking frame, and positioning the position of the three-dimensional rectangular marking frame in the three-dimensional space by using the first three-dimensional point.

And selecting the position of a second anchor point in the display window, calculating the three-dimensional coordinate of a second three-dimensional point corresponding to the second anchor point in the three-dimensional space according to the projection transformation matrix of the camera and the road surface coordinate system information, constraining the second three-dimensional point on one vertex of the three-dimensional rectangular marking frame, and forming a first edge of the three-dimensional rectangular marking frame by the connecting line of the second three-dimensional point and the first three-dimensional point. And determining the size of the three-dimensional rectangular frame in the direction of the first edge by using the first edge.

And selecting the position of a third anchor point in the display window, calculating the three-dimensional coordinate of a third three-dimensional point corresponding to the third anchor point in the three-dimensional space according to the camera projection transformation matrix and the road surface coordinate system information, constraining the third three-dimensional point on one vertex of the three-dimensional rectangular marking frame, and forming a second edge of the three-dimensional rectangular marking frame by the connecting line of the third three-dimensional point and the second three-dimensional point. And constraining the second edges to be perpendicular to the first edges in three-dimensional space. And determining the dimension of the three-dimensional rectangular marking frame in the direction of the second edge by using the second edge.

And selecting the position of a fourth anchor point in the display window, calculating the three-dimensional coordinate of a fourth three-dimensional point corresponding to the fourth anchor point in the three-dimensional space according to the camera projection transformation matrix and the road surface coordinate system information, constraining the fourth three-dimensional point on one vertex of the three-dimensional rectangular marking frame, and forming a third edge of the three-dimensional rectangular marking frame by the connecting line of the fourth three-dimensional point and the third three-dimensional point. And constraining the third edge to be perpendicular to the first edge and the second edge in the three-dimensional space, and determining the dimension of the three-dimensional rectangular marking frame in the direction of the third edge by using the third edge.

And rotating the three-dimensional rectangular marking frame around an axis in the normal vector direction of the road surface plane by adjusting the value of the attitude angle of the three-dimensional rectangular marking frame, so that the attitude of the three-dimensional rectangular marking frame is consistent with the attitude of the corresponding obstacle in the video image.

And adjusting the position of the three-dimensional rectangular labeling frame in the three-dimensional space by adjusting the position of the first anchor point. And adjusting the size of the three-dimensional rectangular marking frame in the first edge direction by adjusting the position of the second anchor point. And adjusting the size of the three-dimensional rectangular marking frame in the second arris direction by adjusting the position of the third anchor point. And adjusting the size of the three-dimensional rectangular marking frame in the third edge direction by adjusting the position of the fourth anchor point.

According to the camera projection transformation information and the road surface coordinate system information, all edges of the three-dimensional rectangular marking frame are displayed on the display window, and the size of the three-dimensional rectangular marking frame can be adjusted by moving the positions of the edges in the display window.

And marking the position, the length, the width, the height and the attitude angle of each obstacle in the three-dimensional space by using the corresponding three-dimensional rectangular marking frame in the video image. And after the three-dimensional information of the video image is labeled, storing the three-dimensional labeling information into a storage device, and/or sending the three-dimensional labeling information to a target device through a transmission device.

According to the same technical concept, the present application provides an obstacle information labeling apparatus, as shown in fig. 11, specifically as follows:

the first obtaining module 121 is configured to obtain a video image collected by a camera, a projection transformation matrix of the camera, and a road coordinate system corresponding to a scene shot by the camera;

a first determining module 122 for determining a target obstacle in the video image;

the anchor point obtaining module 123 is configured to, when the target obstacle does not have initialized three-dimensional labeling information, sequentially obtain 4 anchor points that satisfy geometric constraint conditions according to a three-dimensional coordinate system where the camera is located and a road coordinate system corresponding to a scene that the camera takes a picture; the geometrical constraint conditions are as follows: three line segments corresponding to 3 line segments formed by the 4 anchor points according to the acquisition sequence in a three-dimensional space are mutually vertical, so that a three-dimensional rectangular space area occupied by the target barrier is consistent with a three-dimensional rectangular space area occupied by a three-dimensional rectangular marking frame formed by the 4 anchor points;

the second obtaining module 124 is configured to initialize the attitude angle of the three-dimensional rectangular labeling frame as an initial attitude angle; obtaining initial three-dimensional coordinates corresponding to the 4 anchor points according to the initial two-dimensional coordinates of the 4 anchor points in the video image, the projective transformation matrix of the camera, the road coordinate system information of the scene shot by the camera and the initial attitude angle of the three-dimensional rectangular marking frame; wherein, the attitude angle of the three-dimensional rectangular marking frame is as follows: an included angle between the forward direction of the three-dimensional rectangular marking frame and the forward direction of the road surface coordinate system;

the first labeling module 125 is configured to determine three-dimensional labeling information of the target obstacle according to the initial three-dimensional coordinates corresponding to the 4 anchor points, and the attitude angles of the three-dimensional rectangular labeling frame and the three-dimensional rectangular labeling frame, and label the three-dimensional labeling information of the target obstacle in the video image. The first labeling module comprises: the first determining submodule is used for determining the positions and the sizes of the three-dimensional rectangular marking frames where the 4 anchor points are located according to the initial three-dimensional coordinates corresponding to the 4 anchor points respectively;

the second determining submodule is used for determining the three-dimensional labeling information of the target obstacle according to the initial three-dimensional coordinates, the position and the size of the three-dimensional rectangular labeling frame and the initial attitude angle of the three-dimensional rectangular labeling frame which are respectively corresponding to the 4 anchor points under the condition that the initial attitude angle of the three-dimensional rectangular labeling frame is the same as the attitude angle of the target obstacle; wherein the attitude angle of the target obstacle is: an angle between a forward direction of the target obstacle and a forward direction of the road coordinate system.

The attitude angle adjusting module is used for adjusting the initial attitude angle of the three-dimensional rectangular marking frame under the condition that the initial attitude angle of the three-dimensional rectangular marking frame is different from the attitude angle of the target obstacle, so that the adjusted attitude angle of the three-dimensional rectangular marking frame is the same as the attitude angle of the target obstacle;

the second determining module is used for determining the adjusted two-dimensional coordinates of the 4 anchor points according to the adjusted attitude angle of the three-dimensional rectangular marking frame;

the third determining module is used for determining the adjusted three-dimensional coordinates corresponding to the adjusted attitude angles of the 4 anchor points according to the adjusted two-dimensional coordinates of the 4 anchor points, the projection transformation matrix of the camera, the road coordinate system corresponding to the scene shot by the camera and the adjusted attitude angles of the three-dimensional rectangular marking frame;

and the second labeling module is used for determining the three-dimensional labeling information of the target obstacle according to the adjusted three-dimensional coordinates, the position and the size of the three-dimensional rectangular labeling frame and the posture angle of the adjusted three-dimensional rectangular labeling frame which are respectively corresponding to the 4 anchor points, and labeling the three-dimensional labeling information of the target obstacle in the video image.

The third acquisition module is used for acquiring an initialized three-dimensional rectangular marking frame according to the initialized three-dimensional marking information under the condition that the target obstacle has the initialized three-dimensional marking information;

and the third labeling module is used for labeling the initialized three-dimensional labeling information in the video image under the condition that the three-dimensional rectangular space area occupied by the target obstacle is consistent with the three-dimensional rectangular space area occupied by the initialized three-dimensional rectangular labeling frame.

A new anchor point determining module, configured to, when the three-dimensional rectangular space area occupied by the target obstacle is not consistent with the three-dimensional rectangular space area occupied by the initialized three-dimensional rectangular labeling frame, adjust positions of at least 1 anchor point in the initialized three-dimensional rectangular labeling frame and/or an attitude angle of the three-dimensional rectangular labeling frame, so that the three-dimensional rectangular space area occupied by the target obstacle is consistent with the three-dimensional rectangular space area occupied by a new three-dimensional rectangular labeling frame formed by new positions of the adjusted 4 anchor points;

a fourth obtaining module, configured to obtain initial three-dimensional coordinates corresponding to the adjusted new positions of the 4 anchor points according to the initial two-dimensional coordinates of the adjusted new positions of the 4 anchor points in the video image, the projective transformation matrix of the camera, and the road coordinate system information corresponding to the camera;

and the fourth labeling module is used for determining new three-dimensional labeling information of the target obstacle according to the initial three-dimensional coordinates, the new three-dimensional rectangular labeling frame and the attitude angle of the new three-dimensional rectangular labeling frame which correspond to the new positions of the 4 anchor points respectively, and labeling the new three-dimensional labeling information of the target obstacle in the video image.

Wherein, three-dimensional rectangle mark frame acquisition module includes:

the first acquisition submodule is used for acquiring a video image acquired by a camera, a projection transformation matrix of the camera and a road coordinate system corresponding to a scene shot by the camera;

a third determining submodule for determining a target obstacle in the video image;

the estimation submodule is used for estimating the attitude angle of the target obstacle to obtain the estimated attitude angle of the target obstacle; wherein the attitude angle of the target obstacle is: an angle between a forward direction of the target obstacle and a forward direction of a road surface coordinate system;

the second acquisition submodule is used for sequentially acquiring 4 initial anchor points meeting geometric constraint conditions according to the estimated attitude angle of the target obstacle, the three-dimensional coordinate system where the camera is located and the road coordinate system corresponding to the scene shot by the camera; the geometrical constraint conditions are as follows: three line segments corresponding to 3 line segments formed by the 4 initial anchor points according to the acquisition sequence in a three-dimensional space are mutually vertical, so that a three-dimensional rectangular space area occupied by the target barrier is consistent with a three-dimensional rectangular space area occupied by an initial three-dimensional rectangular labeling frame formed by the 4 initial anchor points;

the initial three-dimensional coordinate submodule is used for obtaining initial three-dimensional coordinates corresponding to the 4 initial anchor points according to the initial two-dimensional coordinates of the 4 initial anchor points in the video image, a projection transformation matrix of the camera, a road coordinate system corresponding to a scene shot by the camera and an estimated attitude angle of the target obstacle;

a fourth determining submodule, configured to determine initialized three-dimensional labeling information of the target obstacle according to the initial three-dimensional coordinates, the position and the size of the initial three-dimensional rectangular labeling frame, and the estimated attitude angle, which correspond to the 4 initial anchor points, respectively;

and the marking submodule is used for marking the initialized three-dimensional marking information of the target obstacle in the video image.

The present application also provides an electronic device, as shown in fig. 11, including:

a processor 111;

a memory 112 for storing the processor-executable instructions;

wherein the processor 111 is configured to execute operations performed to implement an obstacle information labeling method.

The present application also provides a non-transitory computer-readable storage medium, in which instructions, when executed by a processor of a terminal, enable the terminal to perform operations performed to implement an obstacle information labeling method.

In addition, as shown in fig. 12, a system for annotating three-dimensional information of a video image on which the method for annotating obstacle information provided by the present application can depend includes: display device, interactive device, computing device, information storage device, information transmission device.

The display device is used for displaying a video image and corresponding three-dimensional labeling information in a display window, the interaction device is used for selecting anchor points in the display window, coordinates of three-dimensional points corresponding to each anchor point in a three-dimensional space are calculated through the calculation device, and according to the three-dimensional coordinate information corresponding to the selected 4 anchor points, three-dimensional space position information, three-dimensional rectangular frame attitude information and length, width and height information of the three-dimensional rectangular frame corresponding to the three-dimensional rectangular labeling frame are calculated, so that the three-dimensional labeling information of the target obstacle is obtained. And once the three-dimensional information annotation of the obstacle in the video image is finished, storing the three-dimensional annotation information in the video image into the information storage device and/or sending the three-dimensional annotation information out through the information transmission device.

For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.

The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.

The method, the device, the electronic device and the storage medium for labeling the obstacle information provided by the present application are introduced in detail, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the above embodiment is only used to help understand the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. An obstacle information labeling method, characterized by comprising:

determining a target obstacle in the video image;

and determining the three-dimensional labeling information of the target obstacle according to the initial three-dimensional coordinates, the three-dimensional rectangular labeling frame and the attitude angle of the three-dimensional rectangular labeling frame corresponding to the 4 anchor points, and labeling the three-dimensional labeling information of the target obstacle in the video image.

2. The method according to claim 1, wherein determining three-dimensional labeling information of the target obstacle according to the initial three-dimensional coordinates, the three-dimensional rectangular labeling frame, and attitude angles of the three-dimensional rectangular labeling frame corresponding to the 4 anchor points respectively comprises:

determining the positions and the sizes of the three-dimensional rectangular marking frames where the 4 anchor points are located according to the initial three-dimensional coordinates corresponding to the 4 anchor points respectively;

under the condition that the initial attitude angle of the three-dimensional rectangular marking frame is the same as the attitude angle of the target obstacle, determining three-dimensional marking information of the target obstacle according to the initial three-dimensional coordinates corresponding to the 4 anchor points, the position and the size of the three-dimensional rectangular marking frame and the initial attitude angle of the three-dimensional rectangular marking frame; wherein the attitude angle of the target obstacle is: an angle between a forward direction of the target obstacle and a forward direction of the road coordinate system.

3. The method of claim 2, further comprising:

under the condition that the initial attitude angle of the three-dimensional rectangular marking frame is different from the attitude angle of the target obstacle, adjusting the initial attitude angle of the three-dimensional rectangular marking frame to enable the adjusted attitude angle of the three-dimensional rectangular marking frame to be the same as the attitude angle of the target obstacle;

4. The method of claim 1, further comprising:

under the condition that the target obstacle has initialized three-dimensional labeling information, acquiring an initialized three-dimensional rectangular labeling frame according to the initialized three-dimensional labeling information;

and under the condition that the three-dimensional rectangular space area occupied by the target obstacle is consistent with the three-dimensional rectangular space area occupied by the initialized three-dimensional rectangular labeling frame, labeling the initialized three-dimensional labeling information in the video image.

5. The method of claim 4, further comprising:

under the condition that the three-dimensional rectangular space area occupied by the target obstacle is not consistent with the three-dimensional rectangular space area occupied by the initialized three-dimensional rectangular marking frame, adjusting the positions of at least 1 anchor point in the initialized three-dimensional rectangular marking frame and/or the attitude angle of the three-dimensional rectangular marking frame, so that the three-dimensional rectangular space area occupied by the target obstacle is consistent with the three-dimensional rectangular space area occupied by a new three-dimensional rectangular marking frame formed by the new positions of the adjusted 4 anchor points;

obtaining initial three-dimensional coordinates corresponding to the adjusted new positions of the 4 anchor points according to the initial two-dimensional coordinates of the adjusted new positions of the 4 anchor points in the video image, the projection transformation matrix of the camera and the road surface coordinate system information corresponding to the camera;

and determining new three-dimensional labeling information of the target obstacle according to the initial three-dimensional coordinates, the new three-dimensional rectangular labeling frame and the attitude angle of the new three-dimensional rectangular labeling frame which correspond to the new positions of the 4 anchor points respectively, and labeling the new three-dimensional labeling information of the target obstacle in the video image.

6. The method of claim 4, wherein the initialized three-dimensional labeling information is obtained as follows:

acquiring a video image acquired by a camera, a projection transformation matrix of the camera and road surface coordinate system information corresponding to a scene shot by the camera;

determining a target obstacle in the video image;

estimating the attitude angle of the target obstacle according to the orientation of the target obstacle to obtain the estimated attitude angle of the target obstacle; wherein the attitude angle of the target obstacle is: an angle between a forward direction of the target obstacle and a forward direction of a road surface coordinate system;

sequentially acquiring 4 initial anchor points meeting geometric constraint conditions according to the estimated attitude angle of the target obstacle, the three-dimensional coordinate system where the camera is located and the road coordinate system corresponding to the scene shot by the camera; the geometrical constraint conditions are as follows: three line segments corresponding to 3 line segments formed by the 4 initial anchor points according to the acquisition sequence in a three-dimensional space are mutually vertical, so that a three-dimensional rectangular space area occupied by the target barrier is consistent with a three-dimensional rectangular space area occupied by an initial three-dimensional rectangular labeling frame formed by the 4 initial anchor points;

obtaining initial three-dimensional coordinates corresponding to the 4 initial anchor points according to the initial two-dimensional coordinates of the 4 initial anchor points in the video image, a projection transformation matrix of the camera, a road coordinate system corresponding to a scene shot by the camera and an estimated attitude angle of the target obstacle;

determining initialized three-dimensional labeling information of the target obstacle according to the initial three-dimensional coordinates corresponding to the 4 initial anchor points, the position and the size of the initial three-dimensional rectangular labeling frame and the estimated attitude angle;

the method further comprises the following steps:

and marking the initialized three-dimensional marking information of the target obstacle in the video image.

7. An obstacle information labeling apparatus, characterized in that the apparatus comprises:

8. The apparatus of claim 7, wherein the first labeling module comprises:

the first determining submodule is used for determining the positions and the sizes of the three-dimensional rectangular marking frames where the 4 anchor points are located according to the initial three-dimensional coordinates corresponding to the 4 anchor points respectively;

9. An electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute to implement an obstacle information labeling method as claimed in any one of claims 1 to 6.

10. A non-transitory computer readable storage medium, wherein instructions, when executed by a processor of a terminal, enable the terminal to perform a method of implementing an obstacle information annotation process according to any one of claims 1 to 6.