CN112447060A

CN112447060A - Method and device for recognizing lane and computing equipment

Info

Publication number: CN112447060A
Application number: CN201911315389.8A
Authority: CN
Inventors: 李冬虎; 吕志畅; 吕跃强
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2019-08-28
Filing date: 2019-12-18
Publication date: 2021-03-05

Abstract

The application provides a method, a device and computing equipment for recognizing lanes, and belongs to the technical field of intelligent traffic. The method comprises the following steps: when lanes are identified in the video, acquiring a video shot by monitoring equipment arranged on the traffic road, recording a plurality of vehicles running on the traffic road in the video, determining the position of each vehicle in a plurality of video frames of the video, determining the driving track of each vehicle in the video according to the position of each vehicle in the plurality of video frames of the video, and identifying at least one lane in the video according to the driving tracks of the vehicles in the video. By the method and the device, the lane in the video can be dynamically identified, and lane identification accuracy is improved.

Description

Method and device for recognizing lane and computing equipment

The present application claims priority from chinese patent application No. 201910804345.5 entitled "a method, apparatus and device for identifying lane lines in video" filed on 28.08/2019, which is incorporated herein by reference in its entirety.

Technical Field

The present application relates to the field of intelligent transportation (intelligent transportation), and in particular, to a method, an apparatus, and a computing device for recognizing a lane.

Background

With the development of intelligent transportation, a large number of monitoring devices are often adopted to monitor the traffic state of a traffic road at present, and then the traffic state of the traffic road is analyzed, regulated and controlled according to videos recorded by the monitoring devices. When analyzing and controlling the traffic state by using the videos recorded by the monitoring devices, it is usually necessary to obtain the positions of the lane lines on the traffic road in the videos shot by each monitoring device, and further determine the positions of the lanes in the videos. Based on the location of the lanes in the video, traffic event analysis may be performed, e.g., traffic flow analysis, traffic violation event determination, etc. The accuracy of determining the position of the lane in the video directly affects the accuracy of subsequent traffic events and traffic flow analysis, so a determination mode with higher accuracy is needed.

In the related art, the video images shot by each camera are observed manually in advance, the position of the lane line in the video is marked in the video images shot by each camera (namely the pixel position of the lane line in the video shot by each camera is determined in advance according to the video images), and the position of the manually marked lane line in the video is stored by the system. In the subsequent analysis process of the traffic incident, after the video shot by the monitoring equipment is obtained, the traffic incident and the like can be determined according to the position of the lane in the video, which is calibrated in advance manually. However, in some cases (for example, when the shooting angle of the monitoring device changes), since it cannot be timely found that the shooting angle of the monitoring device has changed, when the lane calibrated manually is applied to the video shot after the shooting angle changes, the actual positions of the lane calibrated in the video and the current lane in the video do not coincide with each other, and further, the traffic incident analysis result is inaccurate.

Disclosure of Invention

The application provides a method, a device and a computing device for recognizing lanes, which are used for dynamically recognizing the lanes in a video and improving the accuracy of lane recognition.

In a first aspect, the present application provides a method of recognizing a lane, which may be performed by a recognition device. Specifically, the recognition device may obtain a video shot by a monitoring device disposed on the traffic road, and the video records vehicles passing through a shooting range of the monitoring device on the traffic road. The recognition device then inputs the video to the vehicle detection model, and obtains the position of each vehicle in a plurality of video frames of the video (the position is the pixel position of the vehicle in the video frames). The recognition device determines the driving track of each vehicle in the video by using the position of each vehicle in a plurality of video frames of the video. The recognition device then recognizes at least one lane in the video using the trajectories of the plurality of vehicles in the video.

The method can dynamically determine the lanes in the video according to the video without manually determining the lanes in the video in advance by observing the video images. Even if the shooting angle of the monitoring equipment changes, the recognition device can acquire the lane in the video with the changed angle in time, and the accuracy of the traffic incident analysis result can be improved.

In a possible implementation manner, the recognition device may input the video to the vehicle type detection model, and obtain the type of each vehicle in the video, where the type of the vehicle may include a bus, a car, a bus, and the like. The recognition means may determine the trajectories included in each lane, and then, for any lane recognized, the recognition means may determine the type of travel of the lane using the trajectories included in each lane and the type of vehicle to which the trajectories included in the lane belong. For any lane, the type of travel of that lane is used to indicate the types of vehicles that may be traveling on that lane.

In this way, the driving type of the recognized lane in the video is further recognized, and reference is provided for judging whether the vehicle drives on the lane capable of driving.

In a possible implementation manner, the video acquired by the identification device is a video before entering the intersection, the traffic road shot by the monitoring device includes the intersection, the identification device can acquire an extended video shot by the monitoring device, the extended video is a video shot by the monitoring device at a later period of time after the video is shot, and then the extended video is a video of the vehicle passing through the intersection. The recognition device may input the extension video to the vehicle detection model, obtaining a position of each vehicle in a plurality of video frames of the extension video. The recognition device determines the trajectory of each vehicle in the video using the position of each vehicle in the plurality of video frames of the extended video. Then, the recognition device can determine the driving tracks belonging to the same vehicle according to the attributes (such as license plate numbers) of the vehicles belonging to the driving tracks of the plurality of vehicles in the extended video and the attributes of the vehicles belonging to the driving tracks obtained in the video. The identification device determines the driving track of the vehicle at the intersection in the driving tracks belonging to the same vehicle as the prolonged driving track of the driving track obtained in the video. And then the identification device determines the attribute of each lane according to the driving tracks of the vehicles in the video and the extended driving tracks of the vehicles in the extended video.

The attributes of each lane may include any one of the following attributes and combinations thereof: right-turn lanes, left-turn lanes, through-lanes, right-turn through-lanes, left-turn through-lanes, and hybrid lanes. The right-turn lane is a lane only used for turning the vehicle to the right; the left-turn lane is a lane only used for turning the vehicle to the left; the straight lane is a lane only used for the vehicle to run towards the front; the right-turn straight lane is a lane for turning the vehicle to the right and driving the vehicle forwards; the left-turn straight lane is a lane for turning left and driving forward; the hybrid lane is a lane for turning the vehicle to the right, turning the vehicle to the left, and traveling forward.

Thus, the attribute of the lane is recognized, and reference can be provided for judging whether the vehicle runs on the correct lane when passing through the intersection.

In a possible implementation manner, for any one of the trajectories in the video, assuming that the trajectory is the trajectory of the first vehicle, the identifying device may determine the closest lane between the trajectory of the first vehicle and the identified lane, that is, the adjacent lane for acquiring the trajectory of the first vehicle. The recognition device then determines the distance between the track of the first vehicle and the adjacent lane, and determines the distance and the pixel width of the first vehicle in the video (referred to as the pixel width of the first vehicle). If the distance between the driving track of the first vehicle and the adjacent lane is greater than the pixel width of the first vehicle, the identification device may determine a new lane according to the driving track of the first vehicle, where the new lane is a lane identified in the video. In addition, if the distance between the trajectory of the first vehicle and the adjacent lane is less than or equal to the pixel width of the first vehicle, the recognition device may determine that the trajectory of the first vehicle belongs to the adjacent lane of the trajectory of the first vehicle.

In this way, for the track of any vehicle, a new lane can be identified by directly using the pixel width of the vehicle and the distance between the track of the vehicle and the adjacent lane of the track of the vehicle, so that the lane in the video can be quickly identified.

In a possible implementation manner, after determining the lanes in the video, the recognition device may determine the trajectories belonging to each lane, and further determine the number of trajectories of each lane. The recognition device determines the number of the driving tracks in a period of time of each lane as the flow of each lane in the video, and provides reference for adjusting the time of traffic indicator lights at intersections and the like.

In one possible implementation, after recognizing the lanes in the video, the recognition device may obtain an actual number (i.e., a preset number) of the lanes detected in the video. And then the identification device can judge the number of the identified lanes and the preset number, when the number of the identified lanes is not equal to the preset number, the identification device inputs the driving tracks of the plurality of vehicles in the video and the preset number into an aggregation processing algorithm, and outputs the preset number of the driving tracks obtained by aggregating the driving tracks of the plurality of vehicles in the video. The recognition device determines a lane line of the corrected lane according to the two adjacent driving tracks, and obtains at least one corrected lane in the video. Alternatively, the recognition device acquires lane recognition lines (straight lines for marking lanes to be recognized), and determines the intersection of each of the trajectories of the plurality of vehicles in the video with the lane recognition lines. And then the identification device inputs the coordinates and the preset number of all the intersection points into an aggregation processing algorithm, and the output is the preset number of intersection points. The recognition device determines the center point of each intersection point, and determines the straight line perpendicular to the lane recognition line in the straight line where each center point is located as the center line of the lane. And then determining the central lines of the two adjacent central lines as lane lines of the lane, wherein for the lane to which the left central line belongs, the central line is the right lane line of the lane, and for the lane to which the right central line belongs, the central line is the left lane line of the lane. In this way, the recognition device may obtain at least one corrected lane in the video.

Therefore, when the recognized lane is inaccurate, the lane in the video can be corrected, and the accuracy of the finally recognized lane is improved.

In one possible implementation, before recognizing the lane, the worker may mark a lane recognition line in the video frame of the video, which may cover the lane to be detected. Or after the identification device acquires the video, a line segment which is in a video frame of the video and is a preset distance away from the bottom of the video frame and parallel to the bottom of the video frame can be determined as a lane identification line, and the length of the line segment is a preset length. The recognition device may acquire the lane recognition line and then determine a straight line perpendicular to the lane recognition line and intersecting with a left end point of the lane recognition line as a left boundary line of the area to be recognized, and the recognition device may determine a straight line perpendicular to the lane recognition line and intersecting with a right end point of the lane recognition line as a right boundary line of the area to be recognized. In this way, the recognition means can determine the area to be recognized. The recognition device can then recognize at least one lane in the area to be recognized in the video according to the driving tracks of the plurality of vehicles in the area to be recognized in the video.

Therefore, when a plurality of lanes are included in the video, only part of the lanes can be identified according to requirements, and the applicability of the scheme of the application is enhanced.

In one possible implementation, the recognition device receives in real time a video stream captured by a monitoring device disposed on a traffic road. In this way, lanes in the video may be identified in real time. Or the identification device periodically acquires videos shot by monitoring equipment arranged on the traffic road. In this way, the lane in the video shot by the camera can be identified according to a piece of historical video. The different methods for acquiring the video can be suitable for different application scenes, so that the applicability of the scheme is stronger.

In a second aspect, the present application provides an apparatus for recognizing a lane, the apparatus including an obtaining module, a determining module and a recognizing module, wherein the obtaining module is configured to obtain a video captured by a monitoring device disposed on a traffic road, and the video records a plurality of vehicles traveling on the traffic road. The determining module is used for determining the position of each vehicle in the plurality of vehicles in the plurality of video frames of the video and determining the driving track of each vehicle in the video according to the position of each vehicle in the plurality of video frames of the video. The identification module is used for identifying at least one lane in the video according to the driving tracks of the vehicles in the video.

Therefore, the recognition device can determine the lanes in the video according to the video without manual predetermination, so that even if the shooting angle of the monitoring equipment changes, the recognition device can acquire the lanes in the video with the changed angle in time, the lane recognition accuracy is improved, and the accuracy of a traffic incident analysis result can be improved.

In a possible implementation manner, the determining module is further configured to determine a type of each vehicle according to a vehicle type detection model; determining a driving type of each identified lane in the video according to the types of the vehicles and the driving tracks of the vehicles in the video, wherein the driving type is used for indicating the types of the vehicles which can drive on each lane.

In a possible implementation manner, the determining module is further configured to obtain an extended video captured by the monitoring device when the traffic road captured by the monitoring device includes an intersection, where the extended video is a video captured by the monitoring device at a later time after the video is captured; determining trajectories of the plurality of vehicles in the extended video; determining the attribute of each identified lane in the video according to the driving tracks of the vehicles in the video and the driving tracks of the vehicles in the extension video, wherein the attribute of each lane comprises any one of the following attributes and the combination thereof: right-turn lanes, left-turn lanes, through-lanes, right-turn through-lanes, left-turn through-lanes, and hybrid lanes.

In a possible implementation manner, the identification module is specifically configured to determine a distance between a driving track of a first vehicle in the video and an adjacent lane, where the adjacent lane is a lane closest to the driving track of the first vehicle in the identified lanes, compare the distance with a pixel width of the first vehicle, determine that the distance is greater than the pixel width of the first vehicle, and determine a new lane according to the driving track of the first vehicle, where the new lane is one of the at least one identified lane.

In a possible implementation manner, the determining module is further configured to count the traffic flow of each lane in the video.

In a possible implementation manner, the identification module is further configured to, when the number of the identified lanes is not equal to a preset number, obtain at least one corrected lane in the video according to the preset number and the trajectory of each vehicle in the video; wherein the preset number is the actual number of lanes detected in the video.

In a possible implementation manner, the determining module is further configured to determine an area to be identified in the video. The identification module is specifically used for identifying at least one lane in the area to be identified in the video according to the driving tracks of the vehicles in the area to be identified in the video.

In a possible implementation manner, the obtaining module is specifically configured to: receiving a video stream shot by monitoring equipment arranged on a traffic road in real time; or, periodically acquiring videos shot by monitoring equipment arranged on the traffic road.

In a third aspect, the present application provides a computing device for recognizing a lane, the computing device comprising a processor and a memory, wherein: the memory has stored therein computer instructions that are executed by the processor to cause the computing device to implement the method of the first aspect and its possible implementations.

In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions in the computer-readable storage medium are executed by a computing device, the computing device is caused to execute the method of the first aspect and its possible implementation manner, or the computing device is caused to implement the functions of the apparatus of the second aspect and its possible implementation manner.

In a fifth aspect, the present application provides a computer program product containing instructions which, when run on a computing device, cause the computing device to perform the method of the first aspect and its possible implementations or cause the computing device to implement the functions of the apparatus of the second aspect and its possible implementations.

Drawings

FIG. 1 is a system architecture diagram for lane recognition provided by an exemplary embodiment of the present application;

FIG. 2 is a system architecture diagram for lane recognition provided by an exemplary embodiment of the present application;

FIG. 3 is a system architecture diagram for lane recognition provided by an exemplary embodiment of the present application;

FIG. 4 is a schematic block diagram of a computing device provided in an exemplary embodiment of the present application;

FIG. 5 is a block diagram illustrating a system for recognizing a lane according to an exemplary embodiment of the present application;

FIG. 6 is a schematic diagram of a training device 600 and a recognition device according to an exemplary embodiment of the present application;

FIG. 7 is a flowchart illustrating a method for identifying a lane according to an exemplary embodiment of the present application;

FIG. 8 is a schematic illustration of a lane-identifying line provided by an exemplary embodiment of the present application;

FIG. 9 is a schematic flow chart illustrating a process for determining a lane to which each trajectory belongs according to an exemplary embodiment of the present application;

FIG. 10 is a schematic diagram of determining a lane to which each trajectory belongs according to an exemplary embodiment of the present application;

FIG. 11 is a schematic illustration of a lane to be corrected provided by an exemplary embodiment of the present application;

FIG. 12 is a schematic illustration of a revised lane provided by an exemplary embodiment of the present application;

FIG. 13 is a schematic illustration of a revised lane provided by an exemplary embodiment of the present application;

FIG. 14 is a schematic diagram of determining a relationship between a driving trajectory and an extension of a boundary line of a region to be identified, provided by an exemplary embodiment of the present application;

fig. 15 is a schematic structural diagram of an apparatus for recognizing a lane according to an exemplary embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

To facilitate understanding of the embodiments of the present application, the following first introduces concepts of the terms referred to in the embodiments of the present application:

1. the lane, in the embodiment of the present application, refers to a portion for a single tandem vehicle to travel in a video taken by a camera provided on a traffic road. The number of lanes and the width of the lanes of different traffic roads shot by the cameras may be different. In the embodiment of the present application, recognizing a lane means recognizing the position of a lane on a traffic road in a video, that is, recognizing a pixel coordinate sequence (or pixel coordinates and direction of the lane in the video) formed by the lane in the video.

2. Lane lines, in the embodiment of the present application, lane lines refer to boundary lines of lanes in the video, and one lane may include a left lane line and a right lane line.

3. And a monitoring device for monitoring the traveling information of the vehicle in the traffic area. In the embodiment of the present application, the monitoring device may be disposed at a certain position of the intersection or the road (e.g., a central position on the road or an edge position of the road), and is used to monitor the vehicles within the shooting range, and the monitoring device may be a device capable of capturing images or videos, such as a camera, or a camera. In particular, the monitoring device may be a gate device for monitoring vehicles passing through a particular location in the traffic area (e.g., a toll booth, a traffic or security checkpoint, an intersection, a road segment, etc.). The monitoring device may also be an electronic police monitoring device, the content of the data recorded by the electronic police monitoring device being similar to the content of the data captured by the gate device.

In a traffic area, monitoring equipment can be arranged only at some intersections, for example, monitoring equipment can be arranged at trunk sections, sections with high possibility of traffic congestion, sections with intensive accidents and key intersections in the traffic area. The view angle (shooting range) of the monitoring device at the intersection can cover all lanes of the intersection, and the monitoring device arranged at the intersection can shoot vehicles passing through all the lanes of the intersection. The view angle (shooting range) of the monitoring device at the intersection can also only cover the lanes in the partial direction of the intersection, and the monitoring device arranged at the intersection can also only shoot the vehicles passing through the lanes in the partial direction of the intersection.

In the traffic field, the determination of the traffic flow in each driving direction at an intersection, the detection of traffic events, and the like all need to mark lanes in a video shot by a monitoring device. For example, when the line-pressing behavior in the traffic violation event is determined, the lane line of the lane in the video may be calibrated, and then it is determined whether the vehicle in the video covers the lane line of the lane, so as to determine whether the vehicle has the line-pressing behavior. In the related technology, lanes are marked in a video manually, and marking information is obtained, wherein the marking information can comprise a pixel coordinate sequence of lane lines of the lanes in the video, width information of each lane and the like, and the marking information is stored, and after a detection device of a subsequent traffic incident obtains the video shot by a monitoring device, the lanes in the video are directly determined according to the marking information, and then the traffic incident and the like are determined. However, under some conditions (such as artificial damage, natural disasters, and failure of a bracket for installing the monitoring device), after the shooting angle of the monitoring device changes, since it cannot be timely found that the shooting angle of the monitoring device has changed, applying the lane marked by the video shot according to the previous shooting angle to the currently shot video may cause the marked lane to be misaligned with the lane actually recorded in the video, and thus may cause inaccurate detection of a traffic incident and detection of traffic flow. It is desirable to provide a way in which lanes in a video can be dynamically identified based on the content of the video.

Before describing the method for recognizing a lane provided by the embodiment of the present application, a system architecture to which the embodiment of the present application is applied is described.

The lane recognition method provided by the embodiment of the application can be used for recognizing lanes in videos. The method of recognizing a lane may be performed by a recognition device. The identification means may be a hardware device, such as a server, a terminal computing device, etc., or may be a software device (such as a set of software programs running on a hardware device).

The identification device is flexible to deploy and can be deployed in a marginal environment. For example, the identifying means may be an edge computing device or a software means running on one or more edge computing devices in an edge environment, the edge environment referring to a data center or a collection of edge computing devices that are closer to the traffic road to be detected, the edge environment comprising one or more edge computing devices, which may be computing-capable roadside devices disposed at the roadside of the traffic road. For example, as shown in fig. 1, the identification apparatus is disposed at a position closer to the intersection, i.e., the identification apparatus is an edge computing device on the roadside, or the identification apparatus is a software apparatus running on the edge computing device on the roadside. The intersection is provided with a monitoring device which can be networked, the monitoring device shoots videos recording the passing of vehicles at the intersection and sends the videos to the identification device through the network. The recognition device recognizes the lane in the video according to the video.

The recognition device can also be deployed in a cloud environment, which is an entity that provides cloud services to users using basic resources in a cloud computing mode. A cloud environment includes a cloud data center that includes a large number of infrastructure resources (including computing resources, storage resources, and network resources) owned by a cloud service provider, which may include a large number of computing devices (e.g., servers), and a cloud service platform. The recognition device may be a server in the cloud data center for recognizing a lane in the video; the recognition device may also be a virtual machine created in the cloud data center for recognizing lanes in the video; the identification means may also be a software means deployed on a server or a virtual machine in the cloud data center for identifying the lane in the video, which may be deployed distributed over a plurality of servers, or distributed over a plurality of virtual machines, or distributed over virtual machines and servers. For example, as shown in fig. 2, the recognition device is deployed in a cloud environment, and a network-enabled monitoring device disposed on the traffic route side transmits a captured video to the recognition device in the cloud environment. The recognition device recognizes the lane in the video according to the video.

The recognition device can be deployed in a cloud data center by a cloud service provider, the cloud service provider abstracts functions provided by the recognition device into a cloud service, and the cloud service platform is used for users to consult and purchase the cloud service. After purchasing the cloud service, the user can use the service for recognizing the lane provided by the recognition device of the cloud data center. The identification device can also be deployed in a computing resource (such as a virtual machine) of a cloud data center rented by the tenant, the tenant purchases a computing resource cloud service provided by a cloud service provider through a cloud service platform, and the identification device is operated in the purchased computing resource, so that the identification device identifies a lane in the video.

When the identification means is a software means, the identification means may be logically divided into a plurality of parts each having a different function (e.g., the identification means may include an acquisition module, a determination module, an identification module, etc.). Several parts of the recognition device can be respectively arranged on different environments or equipment, and the lane recognition function is realized among the parts of the recognition device arranged on the different environments or equipment in a coordinated mode. For example, as shown in FIG. 3, one part of the identification apparatus (e.g., the acquisition module) is deployed on the edge computing device, and another part of the identification apparatus (e.g., the determination module and the identification module) is deployed on the cloud data center (e.g., on a server or a virtual machine of the cloud data center). The method comprises the steps that monitoring equipment arranged on a traffic road sends a shot video to an acquisition module arranged in edge computing equipment, the acquisition module sends the video to a cloud data center, and a determination module and an identification module arranged on the cloud data center analyze the video to obtain lanes in the video. It should be understood that the present application does not impose a limiting limitation on the partitioning of portions of the identification appliance, nor on which environment the identification appliance is specifically deployed. In actual application, adaptive deployment can be performed according to the computing power of each computing device or specific application requirements. It is to be noted that, in an embodiment, the monitoring device may also be an intelligent camera with certain computing capability, and the recognition apparatus may also be deployed in three parts, where one part is deployed in the monitoring device, another part is deployed in the edge computing device, and another part is deployed in the cloud computing device.

When the identification device is a software device, the identification device can also be deployed on one computing device in any environment (cloud environment, edge environment, terminal computing device, etc.); when the identifying means is a hardware device, the identifying means may be a computing device 400 in any environment. Fig. 4 provides a schematic diagram of a computing device 400, and the computing device 400 shown in fig. 4 includes a memory 401, a processor 402, a communication interface 403, and a bus 404. The memory 401, the processor 402 and the communication interface 403 are connected to each other by a bus 404.

The Memory 401 may be a Read Only Memory (ROM), a static Memory device, a dynamic Memory device, or a Random Access Memory (RAM). The memory 401 may store computer instructions, and when the computer instructions stored in the memory 401 are executed by the processor 402, the processor 402 and the communication interface 403 are used to perform a method of recognizing a lane. The memory may also store data, for example, a portion of the memory 401 for storing data required for the method of recognizing a lane, and for storing intermediate data or result data during execution of the program.

The processor 402 may be a general-purpose Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), a Graphics Processing Unit (GPU), or any combination thereof. Processor 402 may include one or more chips, and processor 402 may include an Artificial Intelligence (AI) accelerator, such as a neural Network Processing Unit (NPU).

The communication interface 403 enables communication between the computing device 400 and other devices or communication networks using transceiver modules, such as, but not limited to, transceivers. For example, data necessary for recognizing a lane can be acquired through the communication interface 403.

Bus 404 may include a path that transfers information between components of computing device 400 (e.g., memory 401, processor 402, communication interface 403).

When the recognition means is a software means, the recognition means can also be distributively disposed on a plurality of computers in the same environment or different environments, therefore, the present application also provides a system for recognizing a lane as shown in fig. 5, which can also be referred to as a lane recognizing computing device, which includes a plurality of computers 500, each computer 500 including a memory 501, a processor 502, a communication interface 503 and a bus 504. The memory 501, the processor 502 and the communication interface 503 are connected to each other by a bus 504.

The memory 501 may also be a read-only memory, a static memory device, a dynamic memory device, or a random access memory. The memory 501 may store computer instructions, and when the computer instructions stored in the memory 501 are executed by the processor 502, the processor 502 and the communication interface 503 are used to perform part of the method of recognizing a lane. The memory 501 may also store data sets.

The processor 502 may also be a general purpose central processing unit, an application specific integrated circuit, a graphics processor, or any combination thereof. The processor 502 may include one or more chips. The processor 502 may include an AI accelerator, such as a neural network processor.

The communication interface 503 enables communication between the computer 500 and other devices or communication networks using transceiver modules, such as, but not limited to, transceivers. For example, video may be acquired through the communication interface 503.

Bus 504 may include a path that transfers information between components of computer 500 (e.g., memory 501, processor 502, communication interface 503).

A communication path is established between each of the computers 500 through a communication network. Any one or more of an acquisition module 1510, a determination module 1520, and an identification module 1530, as referred to hereinafter, may be run on each computer 500. Any of the computers 500 may be a computer (e.g., a server) in a cloud data center, or a computer in an edge data center, or a terminal computing device.

An AI model used in the method for recognizing a lane provided in the embodiment of the present application is described below.

The recognition device needs to adopt a trained AI model when executing the method for recognizing lanes provided by the embodiment of the present application, and the AI model is essentially a mathematical algorithm including mathematical parameters and mathematical formulas (or mathematical rules). The AI model includes a plurality of types, and the neural network model is one type of the AI model, and in describing the embodiments of the present application, the neural network model is taken as an example. It should be understood that other AI models can also be used to perform the functions of the neural network model described in the embodiments of the present application, and the present application is not limited thereto.

Neural network models are a class of mathematical computational models that mimic the structure and function of biological neural networks (the central nervous system of animals). A neural network model may include a number of different functional neural network layers, each layer including parameters and computational formulas. Different layers in the neural network model have different names according to different calculation formulas or different functions, for example, a layer for performing convolution calculation is called a convolution layer, and the convolution layer is commonly used for feature extraction of an input signal (such as an image). One neural network model may also be composed of a combination of a plurality of existing neural network models. Neural network models of different structures may be used in different scenarios (e.g., classification, recognition) or provide different effects when used in the same scenario. The neural network model structure specifically includes one or more of the following: the neural network model has different network layers, different sequences of the network layers, and different weights, parameters or calculation formulas in each network layer. There are many different neural network models with higher accuracy for identifying or classifying application scenarios in the industry, wherein some neural network models can be trained by a specific training set and then perform a task alone or in combination with other neural network models (or other functional modules). Some neural network models may also be used directly to perform a task alone or in combination with other neural network models (or other functional modules).

In some embodiments of the present application, performing a method of recognizing a lane requires the use of two different trained neural network models. One is a neural network model that can be used to detect vehicles in a video after training, and is called a vehicle detection model. It should be understood that the vehicle detection model in the embodiment of the present application may be obtained after being trained by any one of a variety of neural network models that have been built in the industry. For example, a one-stage unified real-time object detection (you only look on: unified, real-time object detection, Yolo) model, a single shot multi-box detector (SSD) model, a Regional Convolutional Neural Network (RCNN) model, or a Fast-regional convolutional neural network (Fast-RCNN) model, etc. The neural network model selected for training to form the vehicle detection model may be referred to as an initial vehicle detection model.

In some embodiments of the present application, another neural network model that is needed to perform the method of identifying lanes is: the model for detecting the type of the detected vehicle is referred to as a vehicle type detection model, and the vehicle type detection model may also be obtained by training any one of some neural network models existing in the industry, such as a Convolutional Neural Network (CNN) model, a residual network (rest) model, a dense network (densnet) model, a Visual Geometry Group network (VGGnet) model, and the like. The neural network model selected for training to form the vehicle type detection model may be referred to as an initial vehicle type detection model. It should be understood that a neural network model capable of realizing vehicle detection and vehicle type detection, which is developed in the future industry, may also be used as the vehicle detection model and the vehicle type detection model in the embodiments of the present application, and the present application is not limited thereto.

The vehicle detection model and the vehicle type detection model can be trained by a training device before being used for recognizing the lane, the training device respectively adopts different training sets to train the initial vehicle detection model and the initial vehicle type detection model, and the vehicle detection model and the vehicle type detection model which are trained by the training device can be deployed in a recognition device which is used for determining the driving track of the vehicle and the type of the vehicle.

Fig. 6 provides a schematic diagram of the structure of a training apparatus 600 and a recognition apparatus. The structure and function of the training device 600 and the recognition device are described below with reference to fig. 6, and it should be understood that the embodiments of the present application only exemplarily divide the structure and function modules of the training device 600 and the recognition device, and the present application does not limit the specific division thereof.

The training device 600 is used for training the initial vehicle detection model 601 and the initial vehicle type detection model 602, and two training sets are required for training the initial vehicle detection model 601 and the initial vehicle type detection model 602, which are called a vehicle detection training set and a vehicle type detection training set respectively. The vehicle detection training set and the vehicle type detection training set obtained by the acquisition device are stored in a database. The acquisition device can acquire a plurality of training videos or training images, and the acquired training videos or training images are processed and labeled by a worker or the acquisition device to form a training set. When the acquisition device acquires a plurality of training videos, the acquisition device takes video frames in the training videos as training images, and then processes and marks the training images to construct a training set. When the training device 600 starts to train the initial vehicle detection model 601, the initialization module 603 first initializes parameters of each layer in the initial vehicle detection model 601 (that is, each parameter is given an initial value), and then the training module 602 reads training images in a vehicle detection training set in a database to train the initial vehicle detection model 601 until a loss function in the initial vehicle detection model 601 converges and a loss function value is smaller than a specific threshold or all training images in the vehicle detection training set are used for training, so that the training of the initial vehicle detection model 601 is completed, and a trained vehicle detection model 605 is obtained. Similarly, when the training device 600 starts training the initial vehicle type detection model 602, the initialization module 603 initializes parameters of each layer in the initial vehicle type detection model 602 (that is, each parameter is given an initial value), and then the training module 604 reads training images in the vehicle type detection training set in the database to train the initial vehicle type detection model 602 until the loss function in the initial vehicle type detection model 602 converges and the loss function value is smaller than the specific threshold or all training images in the vehicle type detection training set are used for training, so that the initial vehicle type detection model 602 is trained completely, and the trained vehicle type detection model 606 is obtained.

It should be noted that the vehicle detection model 605 and the vehicle type detection model 606 may also be obtained by two training devices respectively for training, and the vehicle detection model 605 and/or the vehicle type detection model 606 may also not need to be trained by the training device 600, for example, the vehicle detection model 605 and/or the vehicle type detection model 606 is a neural network model that is trained by a third party and has better accuracy for vehicle detection and/or vehicle type detection.

In some embodiments of the present application, it may also be unnecessary for the acquisition device to acquire training images or training videos, nor to construct a vehicle detection training set and/or a vehicle type detection training set. For example, the vehicle detection training set and/or the vehicle type detection training set are obtained directly from a third party. In addition, it should be noted that the training images in the vehicle detection training set in the present application may have the same content as the training images in the vehicle type detection training set but different labels, for example, the capturing device captures 1 ten thousand images including vehicles traveling on each traffic road. When a vehicle detection training set is constructed, the vehicles in the 1 ten thousand images are marked by using a boundary box, and the 1 ten thousand training images marked by the boundary box form the vehicle detection training set. When a vehicle type detection training set is constructed, the vehicles in the 1 ten thousand images are labeled by using boundary boxes, each boundary box correspondingly labels the type (such as vehicle type, vehicle brand and the like) of the vehicle, and the 1 ten thousand training images after the boundary boxes and the type labels form the vehicle type detection training set.

It should be noted that, in an embodiment of the present application, the recognition device may also use only one trained neural network model, which may be referred to as a detection and recognition model, when recognizing the lane, and the detection and recognition model is a model that includes all functions of the vehicle detection model 605 and the vehicle type detection model 606. The detection and recognition model can detect the position of the vehicle, recognize the vehicle and detect the type of the recognized vehicle. The training of the detection and recognition model is the same as the training concept of the initial vehicle detection model 601 and the initial vehicle type detection model 602, and is not described herein again.

The vehicle detection model 605 and the vehicle type detection model 606 trained by the training apparatus 600 can be used for vehicle detection and vehicle type detection, respectively, of video frames in videos captured by the monitoring devices. In one embodiment of the present application, as shown in FIG. 6, a trained vehicle detection model 605 and a vehicle type detection model 606 are deployed to the recognition device.

A method for recognizing a lane in a video, which may be performed by a recognition device, provided by an embodiment of the present application will be described below with reference to fig. 7. As shown in fig. 7, the processing flow of the method is as follows:

in step 701, an identification device acquires a video shot by a monitoring device arranged on a traffic road.

Wherein, the video records the vehicles passing through the shooting range of the monitoring equipment on the traffic road.

In this embodiment, a communication connection is established between the identification device and the monitoring device, and when the monitoring device shoots a video, the monitoring device transmits the video to the identification device in real time, or the monitoring device periodically sends the shot video to the identification device. So that the recognition means can capture the video.

In step 702, the recognition device determines the position of each vehicle in the plurality of video frames of the video, and determines the trajectory of each vehicle in the video according to the position of each vehicle in the plurality of video frames of the video.

In this embodiment, the recognition device may input the acquired video to the vehicle detection model 605, and the vehicle detection model 605 outputs the position of the boundary box where the vehicle is located in each video frame of the video. For any vehicle in the video, the recognition device determines the position of the vehicle in the boundary frame as the position of the vehicle in the video. For any bounding box position, if the bounding box is a rectangular box, the position may be the position coordinates of the top left corner and the bottom right corner of the bounding box in the video frame, or the position may be the pixel coordinates of the center point of the bounding box in the video frame.

Then the recognition device can input the position of the boundary box included in each video frame into a preset multi-target tracking algorithm, after the multi-target tracking algorithm obtains the position of the boundary box, the distance between the two boundary boxes belonging to the two adjacent video frames can be calculated, based on the calculated distance, the boundary boxes in the two video frames are associated, and a plurality of groups of well-associated boundary boxes are obtained. For example, the video frame 1 and the video frame 2 are two adjacent video frames, the video frame 1 includes 3 bounding boxes, the video frame 2 includes 3 bounding boxes, for any bounding box in the video frame 1, the distance between the bounding box and each of the 3 bounding boxes in the video frame 2 can be calculated, and the two bounding boxes to which the minimum distance in the three distances belongs are determined as the associated bounding boxes. Specifically, the distance between the bounding boxes can be represented by the distance between the center positions of the bounding boxes.

Optionally, the recognition device may further input attribute information of the vehicle (the attribute information of the vehicle may include a vehicle type, a license plate number, and the like) in the bounding box included in each video frame into a preset multi-target tracking algorithm, where the attribute information of the vehicle may be detected and obtained by the vehicle detection model 605, or may be detected and obtained by other neural network models specifically used for detecting the attribute of the vehicle. After the multi-target tracking algorithm obtains the attribute information of the vehicles in the boundary frames, the similarity of the attribute information of the vehicles in the two boundary frames belonging to the two adjacent video frames can be calculated, based on the calculated similarity, the boundary frames in the two video frames are associated, and a plurality of groups of well-associated boundary frames are obtained. For example, the video frame 1 and the video frame 2 are two adjacent video frames, the video frame 1 includes 3 bounding boxes, the video frame 2 includes 3 bounding boxes, for any one of the video frames 1, the similarity between the attribute information of the vehicle in the bounding box and the attribute information of the vehicle in the 3 bounding boxes in the video frame 2 can be calculated, and the two bounding boxes to which the highest similarity belongs in the three similarities are determined as the associated bounding boxes. Specifically, the calculation method of the similarity of the attribute information of the vehicle in any two bounding boxes may be: the similarity of each type of attribute information of the vehicles is determined (for example, when the attribute information of the vehicles is license plate numbers, the license plate numbers of the vehicles in the two boundary frames are the same, and the similarity of the two boundary frames is 100% in the attribute information of the license plate numbers), then the similarities of the different types of attribute information are weighted, and the similarity of the attribute information of the vehicles in the boundary frames is obtained.

Each group of well-associated boundary frames can be regarded as a plurality of boundary frames of the same vehicle in a plurality of video frames, so that the identification device can fit the central position of each group of well-associated boundary frames according to the time sequence of the video frames to obtain a plurality of driving tracks. In this way, the identification device can acquire the trajectory of each vehicle in the video.

It should be noted here that the multi-target tracking algorithm may be stored in the recognition device, and the stored multi-target tracking algorithm is obtained when the recognition device determines the driving track of the vehicle in the video, or the multi-target tracking algorithm is stored in other devices, and the multi-target tracking algorithm is called from other devices when the recognition device determines the driving track of the vehicle in the video. The multi-target tracking algorithm may be any target tracking algorithm capable of associating the bounding boxes in the multiple video frames, for example, a kalman filter algorithm, and specifically, after the position of the bounding box in the video or the attribute information of the vehicle included in the video is input into the multi-target tracking algorithm, the multi-target tracking algorithm outputs the well-associated bounding box.

It should be noted that the trajectory determined in step 702 is the position change of the vehicle in each video frame of the video, that is, the trajectory of the vehicle in the video is represented by using a pixel coordinate sequence, rather than the position change on the actual road.

In step 703, the recognition device recognizes at least one lane in the video according to the driving tracks of the vehicles in the video.

In this embodiment, the recognition device may recognize lanes to which each trajectory belongs in the video one by one, and determine at least one lane in the video based on the lane to which each trajectory belongs in the video. How to identify the lane to which the trajectory belongs in the video will be described later.

Or in other embodiments, lanes in the video may also be determined in a driving track aggregation manner. Firstly, a preset number is obtained, wherein the preset number is the actual number of the lanes in the video, and the preset number can be manually input into the identification device in advance. The identification device can cluster the driving tracks of the vehicles in the video into a preset number class through an aggregation processing algorithm, and determine the central line of each type of driving track. The recognition device determines an area between two adjacent center lines as a lane. In this case, the rightmost lane is a region between a right boundary line of a region to be recognized (described later) and a center line adjacent to the right boundary line, and the leftmost lane is a region between a left boundary line of the region to be recognized and a center line adjacent to the left boundary line. Specifically, the left boundary line of the region to be recognized is a line that belongs to the region to be recognized, is closest to the leftmost side of the region to be recognized, and is parallel to the center line adjacent to the left boundary line. The right boundary line of the area to be identified is a line which belongs to the area to be identified, is closest to the rightmost side of the area to be identified, and is parallel to the central line adjacent to the left boundary line.

It should be noted that the lane determined in step 703 is a lane in the video, and the lane is represented by using pixel coordinates, rather than a lane that is actually switched to the road.

Optionally, in some cases, the video frame of the video may include not only the lane to be recognized but also other lanes, so that it is necessary to determine the area to be recognized in the video, and the lane in the area to be recognized is the lane to be recognized. Specifically, there are various methods for determining the region to be identified in the video by the identification apparatus, and several possible ways are given below:

in the first mode, the recognition device acquires lane recognition lines in the video and determines the area to be recognized according to the lane recognition lines. The lane recognition line is a marking line for determining a region to be recognized in the video.

In some embodiments, prior to identifying the lane, the identification device provides a lane identification line setting interface that the worker may trigger to display. In this setting interface, the operator can draw a lane recognition line in the video frame of the video, which can cover the lane to be detected. For example, as shown in fig. 8, the worker marks a lane recognition line in the video frame, the lane recognition line indicating that only lane 1 and lane 2 are detected and lane 3 is not detected. The recognition device may then store information of the lane recognition line in the video, and subsequently, when performing lane recognition, the recognition device may acquire the lane recognition line in the video.

The recognition means may determine a straight line perpendicular to the lane recognition line and intersecting with a left-side end point of the lane recognition line as a left boundary line of the area to be recognized, and the recognition means may determine a straight line perpendicular to the lane recognition line and intersecting with a right-side end point of the lane recognition line as a right boundary line of the area to be recognized. It should be noted here that the left and right sides of the lane recognition line are described in terms of angles along the vehicle traveling direction. In this way, the recognition means can determine the area to be recognized.

The subsequent identification device can acquire a plurality of driving tracks in the area to be identified based on the position information of the area to be identified in the video frame of the video.

In the second mode, the recognition device may determine a preset distance from the bottom of the video frame in the video frame of the video, and a line segment parallel to the bottom of the video frame as the lane recognition line. And determining the area to be recognized according to the lane recognition line.

The preset distance may be preset, for example, the preset distance is 1/4 that is the height of the video frame.

In this embodiment, after the identification device obtains the video, a line segment, which is in a video frame of the video and is a preset distance away from the bottom of the video frame and parallel to the bottom of the video frame, may be determined as a lane identification line, and the length of the line segment is equal to the length of the video frame. Then, the recognition device may determine the region to be recognized by using the lane recognition line (the determination method is described in the first method, and is not described herein again).

Of course, the length of the line segment parallel to the bottom of the video frame may not be equal to the length of the video frame, but may be a preset length, and the preset length is smaller than the length of the video frame.

It should be noted here that the above-mentioned identification device determines the area to be identified as an optional process, and when the area to be identified is already specified in the video, the identification device may not perform the process of this process.

A more specific flow of the foregoing step 703 is described as follows:

in step 703, as shown in fig. 9, the recognition device may recognize the lane in the video according to the following flow:

in step 901, the identification device arbitrarily selects one of the trajectories (which may be referred to as the trajectory of the first vehicle) of the lanes that the identification device does not determine as the current trajectory.

Step 902, the recognition device determines a nearest lane corresponding to the current driving track in an existing lane of the to-be-recognized area of the current video, wherein the nearest lane refers to a lane closest to the current driving track in the existing lane.

In the present embodiment, the recognition means determines a line made up of the center positions of the left lane line and the right lane line of the current existing lane (i.e., the existing lane of the region to be recognized of the current video), which may be referred to as a center line. The recognition device determines coordinates of an intersection point of a current driving track and a lane recognition line, then determines a distance between the intersection point and a center line of a current existing lane (the distance is a pixel distance), and specifically, during calculation, the recognition device determines coordinates of an intersection point of a center line of a current existing lane and a lane recognition line, and then calculates a distance between two intersection points (one of the intersection points is the intersection point of the current driving track and the lane recognition line, and the other one is the intersection point of the center line and the lane recognition line), namely, the distance between the current driving track and the current existing lane. And the recognition device determines the lane to which the minimum distance belongs in the determined distances as the nearest lane corresponding to the current driving track.

In step 903, the recognition device determines whether the target distance is greater than a target width, where the target distance is a distance between the current driving track and a nearest lane corresponding to the current driving track, and the target width is a width of a bounding box where a vehicle to which the current driving track belongs is located.

In this embodiment, the recognition device determines the width of the bounding box (the width is the pixel width) where the vehicle of the current trajectory belongs to, and determines whether the target distance is greater than the target width.

Step 904, if the target distance is not greater than the target width, the recognition device determines that the lane to which the current driving track belongs is the nearest lane corresponding to the current driving track.

Step 905, if the target distance is greater than the target width, the recognition device determines the lane to which the current driving track belongs according to the nearest lane corresponding to the current driving track.

In this embodiment, the recognition device arbitrarily selects one of the trajectories in the closest lane corresponding to the current trajectory, and determines the center lines of the trajectory and the current trajectory. Or the recognition device aggregates the driving tracks included in the nearest lane corresponding to the current driving track to obtain an aggregated driving track, and determines the central lines of the aggregated driving track and the current driving track. The two situations are the situations that the nearest lane comprises a plurality of driving tracks, and if the nearest lane only comprises one driving track, the driving track and the current driving track are directly used to determine the central line. Specifically, the center lines of the two trajectories (trajectory a and trajectory B) can be determined in various ways, and the following three calculation ways are provided but not limited to: in the first mode, the recognition device determines the intersection point of the trajectory A and the lane recognition line and the intersection point of the trajectory B and the lane recognition line, and takes the central points of the two intersection points in the direction of the lane recognition line. Determining a straight line which comprises the central point and is perpendicular to the lane identification line as the central lines of the driving track A and the driving track B, wherein the first mode can be applied to a linear lane; and secondly, the recognition device moves the lane recognition lines at equal intervals along the driving track A and the driving track B, determines intersection points of the driving track A and the driving track B and the lane recognition lines respectively and determines the central points of the two intersection points when the driving track A and the driving track B move for a certain distance. And the identification device connects all the central points to obtain the central lines of the driving track A and the driving track B. And in the third mode, the recognition device fits the driving track A and the driving track B to obtain a track, wherein the track is the central line of the driving track A and the driving track B.

And then the recognition device judges the position relation between the driving track in the nearest lane and the current driving track. If the driving track in the nearest lane is on the left side of the current driving track, the recognition device determines the central line as the right lane line of the nearest lane, and the recognition device determines the central line as the left lane line of the lane to which the current driving track belongs and determines the adjacent lane line on the right side of the lane as the right lane line of the lane to which the current driving track belongs. If the driving track in the nearest lane is on the right side of the current driving track, the result is just opposite to the above result. Therefore, the lane to which the current driving track belongs is added. It should be noted here that, when determining the center line of the closest lane, the left lane line and the right lane line of the closest lane are already used, and the reason why the right lane line of the closest lane is determined here is: the right lane line of the nearest lane originally adopts the right boundary line of the area to be recognized, but not the real lane line of the nearest lane.

In step 906, the recognition device determines whether there is a driving path of the undetermined lane.

In step 907, if the trajectory of the lane to which the vehicle belongs is not determined, the process returns to step 901.

In step 908, if the vehicle trajectory of the undetermined lane does not exist, the recognition device ends the process of determining the lane to which the vehicle trajectory belongs.

After determining the lane to which each driving track belongs in the video, the recognition device may determine the determined lane as the lane in the video. For example, a total of 50 trajectories in the video, 15 trajectories belonging to a first lane, 20 trajectories belonging to a second lane, and 15 trajectories belonging to a third lane, the lanes in the video can be determined as the first lane, the second lane, and the third lane.

For convenience of understanding, the embodiment of the present application further provides an intuitive process for determining lanes as shown in fig. 10, where the trajectory 1 is the first trajectory selected in step 901, and there is only one lane 1 by default. The lane 2 is the second lane selected in step 901, and the lane 2 is added. The lane 3 is the third lane selected in step 901 and is added with the lane 3. The lane 4 is the fourth lane selected in step 901, and the lane 4 is newly added.

It should be noted that, in the flow shown in fig. 9, when the recognition device determines, as the current trajectory, one trajectory arbitrarily selected for the first time from among the trajectories of the lanes to which the recognition device does not determine currently, there is only one lane in the area to be recognized, where lane lines of the lane are a left boundary line of the area to be recognized and a right boundary line of the area to be recognized, and the lane may be directly determined as the lane to which the current trajectory belongs, without performing the processing of steps 902 to 908. Then, the recognition device re-executes the processing from step 901 to step 908 until determining the lane to which the last trajectory belongs, and the recognition device may re-determine the lane to which the first arbitrarily selected trajectory belongs according to the processing from step 902 to step 908.

It should be noted here that the vehicle generally travels in the vicinity of the middle of the lane, and the distance between the travel track and the nearest lane is less than or equal to the width of the boundary box, which indicates that the vehicle to which the boundary box belongs most probably travels in the nearest lane. And if the distance between the driving track and the nearest lane is greater than the width of the boundary box, the fact that the vehicle to which the boundary box belongs is probably not driven in the nearest lane is described.

In addition, in step 904, the target distance is not greater than the target width, which indicates that the current driving track belongs to the nearest lane corresponding to the current driving track, and the current driving track may be added to the history track list of the nearest lane for the subsequent lane correction processing. Here, a certain number of driving trajectories in a current period of time, for example, a certain number of 50, may be stored in the history trajectory list of each lane.

Optionally, the recognition device may further include step 704 after performing step 703 to determine the lane of the video.

Step 704, determining whether the determined lane is correct, and if not, the recognition device corrects the lane.

The corresponding specific correction process may be as follows:

and the recognition device obtains the lane after correction in the video according to the preset number and the driving track of each vehicle in the video under the condition that the number of the determined lanes is not equal to the preset number.

In this embodiment, the identification device may obtain a preset number, and the preset number may be input in advance by a worker, and the identification device stores the preset number. The preset number is the actual number of lanes detected in the video.

The recognition device may compare the number of lanes determined in step 703 with a preset number, and if the number of lanes is not equal to the preset number, it indicates that the lanes determined in step 703 are inaccurate. The recognition device can perform correction processing on the lane of the region to be recognized in various ways, and three possible ways are given as follows:

in the first mode, the recognition device acquires lane recognition lines and determines the intersection point of each driving track and the lane recognition lines in the driving tracks of the vehicles in the video. And then the identification device inputs the coordinates and the preset number of all the intersection points into an aggregation processing algorithm, and the output is the preset number of intersection points. The recognition device determines the center point of each intersection point, and determines the straight line perpendicular to the lane recognition line in the straight line where each center point is located as the center line of the lane. And then determining the central lines of the two adjacent central lines as lane lines of the lane, wherein for the lane to which the left central line belongs, the central line is the right lane line of the lane, and for the lane to which the right central line belongs, the central line is the left lane line of the lane. Here, for the leftmost lane, the left lane line of the lane is a straight line (may be a left boundary line of the area to be recognized) perpendicular to the lane recognition line and intersecting with the leftmost side of the lane recognition line, for the rightmost lane, the right lane line of the lane is a straight line (may be a right boundary line of the area to be recognized) perpendicular to the lane recognition line and intersecting with the rightmost side of the lane recognition line, and the aggregation processing algorithm may be a K-means (K-means) clustering algorithm or the like. In this way, all lanes corrected in the video can be determined.

For example, as shown in fig. 11, the number of lanes recognized in step 703 is 5, but the number of known targets is 4, and before lane correction: trajectory 1 belongs to lane 1, trajectories 2, 3, 4 belong to lane 2, trajectory 5 belongs to lane 3, trajectory 6 belongs to lane 4, and trajectory 7 belongs to lane 5. Fig. 12 shows the lane after correction, where the trajectory 1 belongs to the lane 1, the trajectories 2, 3, and 4 belong to the lane 2, the trajectory 5 belongs to the lane 3, and the trajectories 6 and 7 belong to the lane 4. It should be understood that only a portion of the wheel path is shown in fig. 11 and 12.

In the second mode, the identification device inputs the driving tracks of the plurality of vehicles in the video and the preset number into the aggregation processing algorithm, and outputs the preset number of driving tracks obtained by aggregating the driving tracks of the plurality of vehicles in the video. The recognition device determines the center lines of any two adjacent vehicle tracks (the manner of determining the center lines of the two vehicle tracks is described in the foregoing text, and is not described herein again), and determines the center lines of the two adjacent vehicle tracks as lane lines of a lane, where for the lane to which the left vehicle track belongs, the center line is the right lane line of the lane, and for the lane to which the right vehicle track belongs, the center line is the left lane line of the lane. Here, the polymerization processing algorithm may be any algorithm that can perform curve fitting, such as a least squares method.

In the third mode, the recognition device obtains the lane after correction in the video according to the driving tracks of the plurality of vehicles in the video, the lane determined in the step 703 and the clustering algorithm.

In the implementation, the recognition device acquires the lane recognition lines, and determines the intersection point of each of the trajectories of the vehicles in the video and the lane recognition lines. And then the identification device inputs the coordinates and the preset number of all the intersection points into an aggregation processing algorithm, and the output is the preset number of intersection points. The identification device determines the central point of each intersection point and determines the middle point of the connecting line of the central points of two adjacent intersection points.

For each determined midpoint, the recognition device determines the lane line of the lane closest to the midpoint among the lanes determined in step 703, and determines a line passing through the midpoint and parallel to the lane line as a new lane line corresponding to the midpoint. In this manner, a new lane line corresponding to each midpoint may be determined. The recognition means determines an area between the adjacent new lane lines as a lane. Here, regarding the leftmost lane or the rightmost lane of the to-be-recognized region, the recognition device determines a region between the left boundary line of the to-be-recognized region and the adjacent new lane line as a lane, and determines a region between the right boundary line of the to-be-recognized region and the adjacent new lane line as a lane. In this way, the recognition means enables updating of the lane determined in step 703.

For example, as shown in fig. 11, the number of targets is 4, the number of lanes recognized in step 703 is 5, and before lane correction: trajectory 1 belongs to lane 1, trajectories 2, 3, 4 belong to lane 2, trajectory 5 belongs to lane 3, trajectory 6 belongs to lane 4, and trajectory 7 belongs to lane 5. The 3 midpoints are determined by calculation, the identifying device determines the lane line closest to each midpoint, and then determines a line passing through each midpoint and parallel to the closest lane line as a new lane line. Then, the recognition device combines the left boundary line, the new lane line and the right boundary line of the region to be recognized into lane lines of the lanes in sequence, namely, the lane 1, the lane 2, the lane 3 and the lane 4 after being updated are obtained, as shown in fig. 13. It should be understood that only a portion of the wheel path is shown in FIG. 13.

Thus, when the lane determined in step 703 is inaccurate, the lane may be corrected to obtain an accurate lane. The first mode and the second mode determine the re-determined lane as the lane in the video, and the third mode corrects the determined lane to obtain the lane in the video.

Due to the fact that the video can be obtained in real time and the lane can be determined, the correct lane in the video can be determined after the shooting angle of the monitoring equipment is changed.

Optionally, in a possible implementation manner, after determining the lanes in the video, the recognition device may further determine the driving type of each lane, specifically processing as follows:

the recognition device determines the type of each vehicle according to a vehicle type detection algorithm, and determines the driving type of each lane recognized in the video according to the types of the vehicles and the driving tracks of the vehicles in the video, wherein the driving type of the lane is used for indicating the type of the vehicle which can drive in each lane.

The types of vehicles may include cars, buses, and the like.

In this embodiment, the identification device may further input each video frame in the video into the vehicle type detection model, and acquire the type of the vehicle in each bounding box, that is, acquire the type of each vehicle in the plurality of vehicles in the video.

Then, the recognition device determines the driving tracks included in each lane in the video, for any lane, the recognition device determines the type of the vehicle to which the driving tracks included in the lane belong, if the lane only includes one type of the vehicle, the type is determined as the driving type of the lane, if the lane includes multiple types of vehicles, but the number of the driving tracks of a certain type of the vehicle is far larger than the number of the driving tracks of other vehicles (for example, the proportion of the number of the driving tracks of the type 1 of the vehicle to the number of the total driving tracks is larger than 98 percent, and the like), the driving type of the lane can be determined as the type of the vehicle with larger proportion. For example, a lane includes only the type of bus, which is a bus lane. If the lane comprises a plurality of types of vehicles and the difference between the number of the tracks of any two types of vehicles in the tracks of the types of the vehicles is less than a certain value, the driving type of the lane is determined to be a mixed type, namely, the lane can be driven by various types of vehicles.

Optionally, in a possible implementation manner, the traffic road shot by the monitoring device includes an intersection, and the area to be identified is an area of the intersection in the video. The identification device may obtain an extended video shot by the monitoring device, where the extended video is a video shot by the monitoring device at a later time when the video is shot by the monitoring device, that is, the video shot by the monitoring device at the later time when the video is shot is a video of a vehicle driving to the intersection from the lane of the video shot in step 701. Here, the monitoring device for shooting the extended video is the same as the monitoring device in step 701, which is assumed that the monitoring device in step 701 can shoot the video of the vehicle passing through the intersection. In the case that the monitoring device in step 701 cannot shoot the video of the vehicle passing through the intersection, the identifying device may determine that the monitoring device belongs to the same intersection as the monitoring device in step 701 and can shoot the monitoring device of the vehicle passing through the intersection, and then obtain the video shot by the monitoring device within a period of time after the video shot in step 701 is shot (the period of time is short, such as 1 minute, etc.), and the video is the extension video.

The recognition device may determine the trajectories of the multiple vehicles in the extended video (which may be referred to as extended trajectories subsequently) in the manner shown in fig. 7, and then the recognition device may determine the trajectories belonging to the same vehicle according to the attributes (such as license plate numbers) of the vehicles to which the trajectories of the multiple vehicles in the extended video belong and the attributes of the vehicles to which the trajectories obtained in step 702 belong. The recognition means determines the trajectory of the vehicle at the intersection among the trajectories belonging to the same vehicle as the extended trajectory of the trajectory obtained in step 702. After step 703, the identifying device may further determine an attribute of each lane based on the extended driving trajectory, where the attribute of the lane is used to indicate a direction in which a vehicle driving on the lane can drive, and the specific process is as follows:

the identification device determines the attributes of lanes in the video according to the driving tracks of the vehicles in the video and the driving tracks of the vehicles in the extension video, wherein the attribute of each lane comprises any one of the following attributes: right-turn lanes, left-turn lanes, through-lanes, right-turn through-lanes, left-turn through-lanes, and hybrid lanes.

The right-turn lane is a lane only used for turning the vehicle to the right; the left-turn lane is a lane only used for turning the vehicle to the left; the straight lane is a lane only used for the vehicle to run towards the front; the right-turn straight lane is a lane for turning the vehicle to the right and driving the vehicle forwards; the left-turn straight lane is a lane for turning left and driving forward; the hybrid lane is a lane for turning the vehicle to the right, turning the vehicle to the left, and traveling forward.

In this embodiment, for any lane in the video, the recognition device may obtain an extended trajectory of the trajectory in the lane, then determine whether the extended trajectory of each trajectory belonging to the lane intersects with an extension line of a boundary line of the area to be recognized, and determine the attribute of the lane based on whether the extended trajectory intersects with the extension line of the boundary line of the area to be recognized. For example, as shown in fig. 14, it is shown that the extended trajectory of the trajectory 1 intersects with the extended line of the left boundary line of the area to be recognized, the extended trajectory of the trajectory 2 intersects with the extended line of the right boundary line of the area to be recognized, and the extended trajectory of the trajectory 3 does not intersect with the extended lines of the left boundary line of the area to be recognized and the extended lines of the right boundary line of the area to be recognized.

If the extended trajectory of the trajectory greater than or equal to M% (such as M is 97) in a certain lane intersects with the extended line of the left boundary line of the area to be recognized, the lane is determined to be a left-turn lane. And if the extended driving track of the driving track which is greater than or equal to M% in a certain lane is intersected with the extension line of the right boundary line of the area to be identified, determining that the lane is a right-turn lane. And if the extended driving track of the driving track of a certain lane which is more than or equal to M% does not intersect with the extended line of the left boundary line of the area to be recognized and does not intersect with the extended line of the right boundary line of the area to be recognized, determining that the lane is a straight lane. And if the extended driving track of the partial driving track belonging to a certain lane does not intersect with the extended line of the left boundary line of the area to be recognized, and the extended driving track of the other partial driving track belonging to the lane intersects with the extended line of the right boundary line of the area to be recognized, determining that the lane is a right-turn straight lane. And if the extended driving track of the part of the driving track belonging to a certain lane does not intersect with the extended line of the right boundary line of the area to be recognized, and the extended driving track of the other part of the driving track belonging to the lane intersects with the extended line of the left boundary line of the area to be recognized, determining that the lane is a left-turn straight lane. In addition, if the extended trajectory of the partial trajectory belonging to a certain lane intersects with the extension line of the left boundary line of the area to be recognized, the extended trajectory of the partial trajectory belonging to the certain lane intersects with the extension line of the right boundary line of the area to be recognized, and the extended trajectory of the partial trajectory belonging to the certain lane does not intersect with the extension line of the right boundary line of the area to be recognized and the extension line of the left boundary line, the lane is determined to be a mixed lane, and the mixed lane can be used for vehicle driving in various driving directions. The boundary line of the area to be recognized is actually a lane line of a lane, and the lane is the leftmost lane and the rightmost lane in the area to be recognized. For example, the area to be recognized belongs to an intersection where the vehicle travels north, the left boundary line of the area to be recognized is the left lane line of the leftmost lane of the intersection, and the right boundary line of the area to be recognized is the right lane line of the rightmost lane of the intersection.

It should be noted that, in order to make the determined attribute of the lane more accurate, the identification device may determine the attribute of each lane by using the trajectory of each lane, and the number of the trajectories of each lane is as large as possible, for example, the number is larger than 50.

It should be noted that the above-mentioned identifying device may determine whether the extended trajectory of each trajectory belonging to the lane intersects with the extended line of the boundary line of the area to be identified in various ways, and the following ways may be used but are not limited to:

the recognition device can judge whether the points on the extended trajectory of the trajectory are distributed on both sides of the extension line of the boundary line of the area to be recognized. If the points on the extended driving track of the driving track are distributed on two sides of the extension line of the boundary line of the area to be identified, determining that the extended driving track of the driving track is intersected with the extension line of the boundary line of the area to be identified; and if the points on the extended driving track of the driving track are not distributed on two sides of the extension line of the boundary line of the area to be identified, determining that the extended driving track of the driving track is not intersected with the extension line of the boundary line of the area to be identified.

Optionally, after determining the lane distribution of the to-be-identified region in the video, the subsequent identification device may further determine the traffic flow of the lane in each driving direction, and adjust the duration of the traffic indicator at the intersection by using the attribute of the lane and the traffic flow of each lane.

In a possible implementation manner, after step 703, the recognition device may determine the traffic flow of each lane by:

the recognition device determines the traffic flow of each lane in the video.

In this embodiment, after determining the lanes in the video, the recognition device may determine the trajectories belonging to each lane, further determine the number of trajectories in each lane within a period of time, and determine the number of trajectories in each lane as the flow rate of each lane in the video.

In addition, after determining the lanes, the attributes of the lanes, and the traffic flow of the lanes in the video, the recognition device may send the determined lanes, the attributes of the lanes, and the traffic flow of the lanes to other devices, and the other devices may perform corresponding processing based on the lanes, the attributes of the lanes, and the traffic flow of the lanes in the video. For example, the other device may determine a traffic violation event based on the lanes in the video. As another example, other devices may regulate traffic lights at intersections based on vehicle flow, and the like. As another example, the other devices may determine whether the lane of the bus is occupied by other non-bus vehicles, etc., based on attributes of the lanes in the video.

In addition, in the embodiment of the present application, the recognition device may further determine the driving direction of the vehicle to which the boundary frame belongs by the positions of the boundary frame in the front and back video frames (for example, if the distance from the position of the boundary frame in the back video frame to the top of the video frame is smaller than the distance from the position of the boundary frame in the front video frame to the top of the video frame, it may be determined that the vehicle to which the boundary frame belongs is driving forward, and conversely, it is determined that the vehicle is driving backward), and further may determine the driving direction of the vehicle on the lane. In addition, the identification device can also determine whether the monitoring equipment is installed in the forward direction or the reverse direction. The forward installation refers to that the monitoring equipment shoots the tail of the vehicle, and the reverse installation refers to that the monitoring equipment shoots the head of the vehicle. Specifically, the distance from the position of the bounding box in the next video frame to the top of the video frame is smaller than the distance from the position of the bounding box in the previous video frame to the top of the video frame, so that it can be determined that the monitoring device is installed in the reverse direction, and vice versa.

In addition, in the embodiment of the present application, since the lane is recognized without depending on recognition of the lane line captured in the video frame, the lane can be recognized even when the lane line in the captured video is blurred due to low illumination or bad weather.

In addition, in the embodiment of the present application, the recognition device in step 701 may obtain the video from the monitoring device in real time, so that the lane in the video may be determined in real time.

In the embodiment of the application, the identification device can acquire videos shot by monitoring equipment arranged on a traffic road, the videos record a plurality of vehicles running on the traffic road, and the positions of each vehicle in a plurality of video frames of the videos are determined. And then determining the driving track of each vehicle in the video based on the positions of the vehicles in a plurality of video frames of the video, and finally determining the lane in the video according to the driving track of each vehicle in the video. Therefore, the recognition device can dynamically determine the lanes in the video according to the video without manual predetermination, so that even if the shooting angle of the monitoring equipment changes, the recognition device can acquire the lanes in the video with the changed angle in time, and the accuracy of a traffic incident analysis result can be improved.

Fig. 15 is a structural diagram of a lane recognition device (i.e., a recognition device) according to an embodiment of the present application. The identification means may be implemented as part or all of the identification means by software, hardware or a combination of both. The identification apparatus provided in the embodiment of the present application can implement the processes described in fig. 7 and fig. 9 in the embodiment of the present application, and the identification apparatus includes: an acquisition module 1510, a determination module 1520, and an identification module 1530, wherein:

an obtaining module 1510, configured to obtain a video captured by a monitoring device disposed on a traffic road, where the video records a plurality of vehicles traveling on the traffic road, and specifically may be configured to implement the obtaining function in step 701 and execute an implicit step included in step 701;

a determining module 1520, configured to determine a position of each vehicle in the plurality of vehicles in the plurality of video frames of the video, and determine a trajectory of each vehicle in the video according to the position of each vehicle in the plurality of video frames of the video, and in particular, may be configured to implement the determining function of step 702 and perform the implicit steps included in step 702;

the identifying module 1530 is configured to identify at least one lane in the video according to the trajectories of the vehicles in the video, and may be specifically configured to implement the identifying function of step 703 and execute the implicit step included in step 703.

In one possible implementation, the determining module 1520 is further configured to:

determining the type of each vehicle according to a vehicle type detection model;

determining a driving type of each identified lane in the video according to the types of the vehicles and the driving tracks of the vehicles in the video, wherein the driving type is used for indicating the types of the vehicles which can drive on each lane.

when the traffic road shot by the monitoring equipment comprises an intersection, acquiring an extension video shot by the monitoring equipment, wherein the extension video is a video shot by the monitoring equipment in a period of time after the video is shot;

determining trajectories of the plurality of vehicles in the extended video;

determining the attribute of each identified lane in the video according to the driving tracks of the vehicles in the video and the driving tracks of the vehicles in the extension video, wherein the attribute of each lane comprises any one of the following attributes and the combination thereof: right-turn lanes, left-turn lanes, through-lanes, right-turn through-lanes, left-turn through-lanes, and hybrid lanes.

In a possible implementation manner, the identifying module 1530 is specifically configured to:

determining the distance between the driving track of a first vehicle in the video and an adjacent lane, wherein the adjacent lane is the lane which is the closest to the driving track of the first vehicle in the identified lanes;

comparing the distance to a magnitude of a pixel width of the first vehicle, determining that the distance is greater than the pixel width of the first vehicle;

and determining a new lane according to the driving track of the first vehicle, wherein the new lane is one of the at least one lane which is identified.

and counting the traffic flow of each lane in the video.

In a possible implementation manner, the identifying module 1530 is further configured to:

when the number of the identified lanes is not equal to a preset number, obtaining at least one corrected lane in the video according to the preset number and the driving track of each vehicle in the video;

wherein the preset number is an actual number of lanes detected in the video.

determining a region to be identified in the video;

the identification module 1530 is specifically configured to:

and identifying at least one lane in the area to be identified in the video according to the driving tracks of the vehicles in the area to be identified in the video.

In a possible implementation manner, the obtaining module 1510 is specifically configured to:

receiving a video stream shot by monitoring equipment arranged on the traffic road in real time; alternatively, the first and second electrodes may be,

and periodically acquiring the video shot by monitoring equipment arranged on the traffic road.

In a more specific embodiment, the obtaining module 1510 is used to execute the process of obtaining the video captured by the monitoring device in fig. 7. The determining module 1520 is configured to execute the process of determining the driving trajectory in fig. 7. The recognition module 1530 is used for executing the process of recognizing the lane in the video in fig. 7 and the flow shown in fig. 9.

The division of the modules in the embodiments of the present application is schematic, and only one logic function division is provided, and in actual implementation, there may be another division manner, and in addition, each functional module in each embodiment of the present application may be integrated in one processor, may also exist alone physically, or may also be integrated in one module by two or more modules. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.

The present application also provides a computing device 400 as shown in fig. 4, wherein a processor 402 in the computing device 400 reads a set of computer instructions stored in a memory 401 to execute the aforementioned method for recognizing lanes.

The descriptions of the flows corresponding to the above-mentioned figures have respective emphasis, and for parts not described in detail in a certain flow, reference may be made to the related descriptions of other flows.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product for implementing lane recognition comprises one or more computer program instructions for recognizing lane, which when loaded and executed on a computer implement the procedures described in the embodiments of this application in fig. 7 and 9 or the functions of the apparatus described in the embodiments of this application in fig. 15, in whole or in part.

The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optics, digital subscriber line, or wireless (e.g., infrared, wireless, microwave, etc.) means, the computer readable storage medium stores a readable storage medium of computer program instructions that implement blind spot detection for vehicles, the computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device including one or more integrated servers, data centers, etc. the available medium may be magnetic medium, (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., DVD), or semiconductor media (e.g., SSD).

Claims

1. A method of recognizing a lane, the method comprising:

acquiring a video shot by monitoring equipment arranged on a traffic road, wherein the video records a plurality of vehicles running on the traffic road;

determining a position of each vehicle of the plurality of vehicles in a plurality of video frames of the video;

determining the driving track of each vehicle in the video according to the position of each vehicle in a plurality of video frames of the video;

and identifying at least one lane in the video according to the driving tracks of the vehicles in the video.

2. The method of claim 1, further comprising:

3. The method according to claim 1 or 2, wherein when the traffic road photographed by the monitoring device includes an intersection, the method further comprises:

acquiring an extended video shot by the monitoring equipment, wherein the extended video is a video shot by the monitoring equipment in a period of time after the video is shot;

determining trajectories of the plurality of vehicles in the extended video;

4. The method according to any one of claims 1-3, wherein identifying at least one lane in the video based on the trajectories of the plurality of vehicles in the video comprises:

5. The method according to any one of claims 1-4, further comprising:

and counting the traffic flow of each lane in the video.

6. The method according to any one of claims 1-5, further comprising:

wherein the preset number is an actual number of lanes detected in the video.

7. The method according to any one of claims 1 to 6, wherein the acquiring of the video shot by the monitoring device arranged on the traffic road comprises:

8. An apparatus for recognizing a lane, comprising:

the system comprises an acquisition module, a display module and a control module, wherein the acquisition module is used for acquiring a video shot by monitoring equipment arranged on a traffic road, and the video records a plurality of vehicles running on the traffic road;

a determining module, configured to determine a position of each vehicle in the plurality of vehicles in a plurality of video frames of the video, and determine a trajectory of each vehicle in the video according to the position of each vehicle in the plurality of video frames of the video;

the identification module is used for identifying at least one lane in the video according to the driving tracks of the vehicles in the video.

9. The apparatus of claim 8, wherein the determining module is further configured to:

10. The apparatus of claim 8 or 9, wherein the determining module is further configured to:

determining trajectories of the plurality of vehicles in the extended video;

11. The apparatus according to any one of claims 8 to 10, wherein the identification module is specifically configured to:

12. The apparatus of any of claims 8-11, wherein the determining module is further configured to:

and counting the traffic flow of each lane in the video.

13. The apparatus of any of claims 8-12, wherein the identification module is further configured to:

wherein the preset number is an actual number of lanes detected in the video.

14. The apparatus according to any one of claims 8 to 13, wherein the obtaining module is specifically configured to:

15. A computing device to identify a lane, the computing device comprising a processor and a memory, wherein:

the memory having stored therein computer instructions;

the processor executes the computer instructions to cause the computing device to perform the method of any of claims 1-7.

16. A computer-readable storage medium storing computer instructions which, when executed by a computing device, cause the computing device to perform the method of any of claims 1-7 or to implement the functionality of the apparatus of any of claims 8-14.