WO2021004077A1

WO2021004077A1 - Method and apparatus for detecting blind areas of vehicle

Info

Publication number: WO2021004077A1
Application number: PCT/CN2020/078329
Authority: WO
Inventors: 冷继南; 沈建惠; 常胜; 吕跃强
Original assignee: 华为技术有限公司
Priority date: 2019-07-09
Filing date: 2020-03-07
Publication date: 2021-01-14
Also published as: CN112216097A

Abstract

A method for detecting blind areas of a vehicle, relating to the field of intelligent transportation. The method comprises: receiving video data photographed by a camera provided on a traffic road; determining position information and posture information of the vehicle on the traffic road at the current time point or a future time point according to the video data, wherein the posture information represents a driving direction of the vehicle; further, obtaining blind area information of the vehicle, the blind area information being decided by structural attributes of the vehicle; and finally, determining blind areas of the vehicle on the traffic road at the current time point or the future time point according to the blind area information of the vehicle and the position information and posture information of the vehicle on the traffic road. According to the method, the blind areas of the vehicle on the traffic road at the current time point or the future time point can be timely determined, the probability of occurrence of a dangerous accident in the blind areas is greatly reduced, and the safety of driving is improved.

Description

Method and device for detecting blind area of vehicle

Technical field

This application relates to the field of smart transportation, and in particular to a method and device for detecting the blind area of a vehicle.

Background technique

With the progress of society, all kinds of vehicles can be seen everywhere on traffic roads. For the driver of the vehicle, the observation and grasp of the traffic conditions around the vehicle during the driving process is particularly important for driving safety. However, due to the model and structure of some vehicles, the driver often has blind spots in the surrounding environment during the normal driving of the vehicle, that is, there are certain areas that the driver cannot observe during driving, such as: Trucks, excavators, loading and unloading trucks and other engineering vehicles are relatively large, and the driver's field of vision is blocked by the body, and part of the environment around the body cannot be observed. Due to the blind area of the driving vehicle, the driver may not be able to know the dangerous situation in the blind area (for example: there are dangerous obstacles in the blind area, pedestrians or other vehicles in the blind area), which brings huge problems to both vehicles and pedestrians. Safety hazards.

At present, in order to solve the safety problems of vehicles and pedestrians caused by the blind area of the vehicle, the method of installing a detector on the body is often adopted, so that when a dangerous situation exists in the blind area of the vehicle, the vehicle detector emits an alarm sound. However, this method is often too late for some dangerous situations, and it is even less effective for running vehicles, because the surrounding conditions of the vehicle during the movement are changing at any time, and the detector cannot learn the dangerous situation in time and send it out in time. Warning prompt. Therefore, how to detect the blind area of the vehicle in a more timely manner is a problem that needs to be solved urgently.

Summary of the invention

This application provides a method for detecting the blind area of a vehicle. The method can detect the blind area of a vehicle on a traffic road at the current or future time in a more timely manner, greatly reducing the probability of dangerous accidents in the blind area, and improving the traffic road Safety.

In the first aspect, the present application provides a method for detecting a blind area of a vehicle, and the method is executed by a detecting device. The method includes: receiving video data taken by a camera set on a traffic road, the video data recording the operating conditions of vehicles and other targets on the traffic road at the current moment; after receiving the video data, determining the current time or the future time of the vehicle according to the video data Position information and posture information on the traffic road, where posture information indicates the direction of travel of the vehicle. Further, the blind spot information of the vehicle is acquired, and the blind spot information is determined by the structural attributes of the vehicle. According to the blind spot information of the vehicle, the position information and posture information of the vehicle on the traffic road, the blind spot of the vehicle on the traffic road at the current or future time is determined.

The above method processes and analyzes the video data taken by a camera installed on a traffic road to obtain the position information and posture information of the vehicle, and combines the blind area information of the vehicle to obtain the blind area of the vehicle on the traffic road at the current time or in the future in time. The determined blind area of the vehicle can be timely given to the vehicle or other targets on the traffic road (such as pedestrians, other vehicles) as a reference, so that the vehicle or pedestrian can avoid and adjust in time according to the determined blind area, greatly reducing the probability of dangerous accidents in the blind area , Improve traffic and road safety. Further, the above method can predict the blind area of the vehicle at a future time, so that the vehicle can judge the safety of driving at the future time in advance, and further reduces the probability of dangerous accidents in the blind area.

In a possible implementation of the first aspect, the above method further includes: determining that the vehicle has a blind spot risk based on the video data and the blind spot of the vehicle on the traffic road at the current or future moments, wherein the blind spot danger indicates that the vehicle is in danger. There are other targets in the blind zone on the traffic road; the above method further includes: sending a blind zone warning to the vehicle or other targets in the blind zone.

According to the above-mentioned blind zone, determine whether there are other targets in the blind zone of the vehicle. If there are other targets in the blind zone, it can be considered that other targets in the blind zone may be collided or scratched by the vehicle, and timely warning can be more targeted. Remind vehicles and pedestrians in a dangerous state in the blind spot, further reducing dangerous accidents in the blind spot.

In a possible implementation of the first aspect, the blind zone warning sent to vehicles with blind spot danger or other targets in the blind zone includes warning data, and the warning data includes one or more of the following information: The location and range of the blind spot on the traffic road, the location information of other targets on the traffic road, and the types of other targets. The content in the warning data can be used by the vehicle driver to determine the strategy for avoiding the danger in the blind area according to the warning, avoiding other targets in the blind area more accurately, and further reducing the probability of danger.

In a possible implementation of the first aspect, determining the position information and posture information of the vehicle on the traffic road at a future time according to the video data includes: determining the position information and posture information of the vehicle on the traffic road at the current time according to the video data; According to the vehicle's current location information and posture information on the traffic road, predict the vehicle's location information and posture information on the traffic road in the future. The predicted position information and posture information of the vehicle at the future time can enable the detection device to determine the location and range of the blind zone of the vehicle at the future time.

In a possible implementation of the first aspect, the method further includes: constructing a visual blind spot image according to the blind spot of the vehicle on the traffic road at the current time or in the future; sending the visual blind spot image to the vehicle or other devices.

By sending the determined blind spot in the form of visual blind spot image to the currently driving vehicle or the equipment of the traffic management platform or other vehicles adjacent to the vehicle, the driver or manager of the vehicle can intuitively and quickly determine the blind spot. The location and range reduce the time for the driver and other targets to react to the blind spot, and improve driving safety.

In a possible implementation of the first aspect, the method further includes: calculating the driving speed of the vehicle; and adjusting the blind spot of the vehicle on the traffic road at the current time or in the future according to the driving speed of the vehicle. As the driving speed of the vehicle will affect the inertia and braking distance when the vehicle is driving, adjust the blind area of the vehicle according to the speed. For example, for a high-speed vehicle, expand the range of the front blind area of the vehicle and provide the adjusted blind area for the driver's reference and remind Drivers pay attention to further reduce the risk of blind spots and improve driving safety.

In a possible implementation of the first aspect, the method further includes: sending an adjustment instruction to a vehicle with a blind spot risk, wherein the adjustment instruction instructs the vehicle to adjust a driving route to avoid other targets in the blind zone. The method can plan a new driving route for the driving vehicle according to the determined blind zone or the position of other targets in the determined blind zone, and instruct the vehicle to adjust the shape of the car route, so that the vehicle can reasonably avoid other targets in the blind zone. It further avoids the problem of the driver's untimely response in a panic, and further improves the driving safety of the vehicle. On the other hand, for self-driving vehicles, after receiving the adjustment instruction, the self-driving vehicle can automatically adjust the driving route according to the new driving route in the adjustment instruction, avoid the danger of blind spots, and realize safe automatic driving.

In a possible implementation of the first aspect, the method further includes: determining a high-risk blind area in the blind area.

In a possible implementation of the first aspect, the method further includes: determining a blind spot risk coefficient according to the blind spot information of the vehicle, the blind spot risk coefficient indicating the degree of danger of each blind spot of the vehicle; according to the blind spot risk factor and the blind spot of the vehicle Information and the location information and posture information of the vehicle on the traffic road determine the high-risk blind area of the vehicle on the traffic road at the current time or in the future. According to the blind zone risk coefficient, determining the high-risk blind zone enables the driver to pay more attention to the dangerous situation in the high-risk blind zone.

In a possible implementation of the first aspect, after the high-risk blind zone is determined according to the blind zone risk coefficient, the high-risk blind zone can also be determined based on the video data and the determined high-risk blind zone on the traffic road at the current or future time. Whether there are other targets in the risk blind zone. This method of first determining the high-risk blind zone and then determining the dangerous situation in the high-risk blind zone can save computing resources and avoid bringing many unnecessary danger reminders to the driver and affecting the normal driving of the driver. For example, for some low-risk blind spots, there are other targets, but there is no danger for the vehicle and the targets in the blind zone, so it is not necessary to remind the driver and the targets in the blind zone.

In a possible implementation of the first aspect, the video data includes multiple video streams taken by multiple cameras set at different positions on the traffic road; the video data is used to determine the current or future time of the vehicle on the traffic road. Location information includes: determining the current or future position of the vehicle in multiple video streams according to multiple video streams; determining the current or future position of the vehicle in multiple video streams according to the current or future position of the vehicle in multiple video streams Location information on traffic roads. By combining multiple video streams to jointly determine the location information of the vehicle, the location of the vehicle on the traffic road can be determined more accurately, and the blind area of the vehicle on the traffic road can be determined more accurately. In addition, the use of video data captured by cameras with multiple viewing angles can expand the range of detected traffic roads, so that vehicles under multiple camera viewing angles can be used to determine blind areas in the same way.

In a possible implementation of the first aspect, the blind area information of the vehicle includes: the number of blind areas, the position of each blind area relative to the vehicle, and the shape of the blind area. According to the above-mentioned blind area information and the position and posture of the vehicle on the traffic road, the distribution and range of the blind area of the vehicle on the traffic road can be determined.

In a possible implementation of the first aspect, the blind spot information of the vehicle includes the risk factor of the blind spot; according to the blind spot information of the vehicle and the position information and posture information of the vehicle on the traffic road at the current or future moments, it is determined that the vehicle is in traffic Blind areas on the road, including: determine the high-risk blind area information in which the risk coefficient of the blind area in the blind area information is greater than the preset danger threshold; according to the high-risk blind area information, the current or future position information and posture of the vehicle on the road Information to determine the high-risk blind area on the traffic road at the current time or in the future.

In a possible implementation of the first aspect, the blind spot information of the vehicle includes the risk factor of the blind spot; before determining that the vehicle has a risk of the blind spot, the method further includes: correcting the risk factor of the blind spot according to the blind spot of the vehicle on the traffic road ; Determine the high-risk blind area of the vehicle on the traffic road according to the relationship between the risk coefficient of the revised blind area and the preset dangerous threshold; According to the video data and the determined blind area of the vehicle on the traffic road at the current time or in the future , Determining that the vehicle is in danger of a blind spot includes: determining that the vehicle is in a high-risk blind zone on the traffic road based on the location information of other targets in the blind spot on the traffic road.

In a possible implementation of the first aspect, before determining that the vehicle is in danger of the blind area according to the video data and the determined blind area of the vehicle at the current or future time, the method further includes: The blind zone on the road determines the risk factor of the blind zone of the vehicle; according to the relationship between the risk factor of the blind zone and the preset risk threshold, the high-risk blind zone of the vehicle on the traffic road is determined; the current or future time of the vehicle is determined according to the video data In the blind area on the traffic road, determining that the vehicle has the risk of the blind area includes: determining that the vehicle has a blind area hazard in the high-risk blind area on the traffic road according to the location information of other targets in the blind area on the traffic road.

The above method of determining high-risk blind areas and then performing blind-area hazard detection on high-risk blind areas can save computing resources on the one hand, and on the other hand, it can avoid vehicles and other targets when there are other targets in some blind areas but do not pose a danger. Carry out warnings to improve the accuracy of warnings and avoid frequent warnings to disturb drivers and pedestrians.

In a possible implementation of the first aspect, the method further includes: determining the blind zone information of the vehicle according to the structural attributes of the vehicle, specifically: querying the blind zone information database according to the structural attributes of the vehicle to obtain the structure of the vehicle in the blind zone information database The blind zone information of the vehicle corresponding to the attribute.

In a possible implementation of the first aspect, the structural attributes of the vehicle include the type of the vehicle; query the blind zone information database according to the structural attributes of the vehicle to obtain the blind zone information of the vehicle corresponding to the structural attributes of the vehicle in the blind zone information database, specifically including : Input the structural attributes to the blind zone information database to obtain the blind zone information corresponding to the vehicle of the same type as the vehicle in the blind zone information database;

In a possible implementation of the first aspect, the structural attributes of the vehicle include the length and width information of the vehicle, the type of cab, and the position of the cab; according to the structural attributes of the vehicle, the blind zone information database is queried to obtain the corresponding structural attributes in the blind zone information database. The blind area information of the vehicle specifically includes: inputting the structural attributes of the vehicle into the blind area information database, and obtaining the blind area information corresponding to the vehicle whose length and width information, cab type, and cab position are similar to the vehicle in the blind area information database.

In a possible implementation of the first aspect, the received video data is a real-time video stream, and the position information and posture information of the vehicle on the traffic road at the current or future time are determined according to the video data, which specifically includes: according to the real-time video Determine the position information of the vehicle in the video data at the current time or the future time; according to the preset calibration relationship and the position information of the vehicle in the video data at the current time or the future time, determine the current time or the future time The position information of the vehicle on the traffic road; the position information of the vehicle on the traffic road at the current time or the future time is used to determine the trajectory information of the vehicle on the traffic road; the current time or the future time position is determined according to the trajectory information of the vehicle The posture information of the vehicle is described, and the posture information indicates the traveling direction of the vehicle.

In a second aspect, the present application also provides a detection device, including: a target detection and tracking module for receiving video data, the video data is captured by a camera set on a traffic road; a target positioning module for determining according to the video data The position information of the vehicle on the traffic road at the current time or in the future; the posture determination module is used to determine the posture information of the vehicle on the traffic road at the current time or the future time according to the video data; the blind spot determination module is used to obtain the blind area of the vehicle Information, blind area information is determined by the structural attributes of the vehicle; it is also used to determine the blind area of the vehicle on the traffic road at the current time or in the future based on the vehicle’s blind area information, position information and posture information of the vehicle on the traffic road.

In a possible implementation of the second aspect, the blind spot determination module is also used to determine that the vehicle has a blind spot hazard based on the video data and the determined blind spot of the vehicle on the traffic road at the current time or in the future. The vehicle has other targets in the blind zone on the traffic road; the blind zone warning is sent to the vehicle or other targets in the blind zone.

In a possible implementation of the second aspect, the blind zone warning includes warning data, and the warning data includes one or more of the following information: the position and range of the blind zone on the traffic road where the blind zone danger occurs, and other targets in the traffic Location information on the road and the types of other targets.

In a possible implementation of the second aspect, when the target positioning module is used to determine the position information of the vehicle on the traffic road at a future time according to the video data, it is specifically used to: determine that the vehicle is in the traffic at the current time according to the video data. Position information on the road; predict the position information of the vehicle on the road in the future according to the position information of the vehicle on the road at the current time; the posture determination module is used to determine the posture of the vehicle on the road in the future based on the video data When the information is used, it is specifically used to determine the vehicle's current posture information on the traffic road according to the video data; according to the vehicle's current posture information on the traffic road, predict the vehicle's posture information on the traffic road in the future.

In a possible implementation of the second aspect, the blind spot determination module is further configured to construct a visual blind spot image according to the blind spot of the vehicle on the traffic road at the current time or in the future; and send the visual blind spot image to the vehicle or other devices.

In a possible implementation of the second aspect, the blind spot determination module is also used to calculate the driving speed of the vehicle according to the position information of the vehicle on the traffic road at the current time or in the future; adjust the current time or the vehicle according to the driving speed of the vehicle A blind spot on the traffic road in the future.

In a possible implementation of the second aspect, the blind spot determination module is also used to send an adjustment instruction to a vehicle in danger of the blind spot, where the adjustment instruction instructs the vehicle to adjust the driving route to avoid other targets in the blind zone.

In a possible implementation of the second aspect, the blind spot determination module is further configured to determine a high-risk blind zone in the blind zone.

In a possible implementation of the second aspect, the blind spot determination module is specifically used to determine the blind spot risk coefficient according to the blind spot information of the vehicle. The blind spot risk coefficient indicates the degree of danger of each blind spot of the vehicle; according to the blind spot risk factor and the vehicle's The blind spot information and the position information and posture information of the vehicle on the traffic road determine the high-risk blind spot on the traffic road at the current time or in the future.

In a possible implementation of the second aspect, the blind spot determination module is also used to determine whether there are other targets in the high-risk blind zone based on the video data and the determined high-risk blind zone on the traffic road at the current time or in the future. .

In a possible implementation of the second aspect, the video data includes multiple video streams captured by multiple cameras set at different positions on the traffic road; the target detection and tracking module is also used to determine The location information of the vehicle in multiple video streams at the current or future time; the target positioning module is also used to determine whether the vehicle is on the traffic road at the current or future time based on the location information of the vehicle in multiple video streams at the current or future time Location information.

In a possible implementation of the second aspect, the blind area information of the vehicle includes: the number of blind areas, the position of each blind area relative to the vehicle, and the shape of the blind area.

In a third aspect, the present application also provides an in-vehicle device, the in-vehicle device is installed on a vehicle, and the in-vehicle device is configured to execute the method provided by the first aspect or any one of the possible implementations of the first aspect.

In a fourth aspect, the present application also provides a vehicle. The vehicle includes a storage unit and a processing unit. The storage unit of the vehicle is used to store a set of computer instructions and data sets. The processing unit executes the computer instructions stored in the storage unit, and the processing unit reads The data set of the storage unit is taken, so that the vehicle executes the method provided by the first aspect or any one of the possible implementation manners of the first aspect.

In a fifth aspect, the present application provides a system that includes at least one memory and at least one processor, at least one memory is used to store a set of computer instructions; when at least one processor executes the set of computer instructions, the system Implement the method provided by the first aspect or any one of the possible implementations of the first aspect.

In a sixth aspect, the present application also provides a detection system, which is used to detect the blind area of a vehicle, and the system includes:

The vehicle dynamic monitoring system is used to receive video data, and determine the position information and posture information of the vehicle on the traffic road at the current time or in the future based on the video data, wherein the video data is captured by a camera set on the traffic road;

The vehicle blind spot detection system is used to obtain the blind spot information determined by the structural attributes of the vehicle, and determine the blind spot of the vehicle on the traffic road at the current time or in the future according to the blind spot information of the vehicle, the position information and posture information of the vehicle on the traffic road.

In a seventh aspect, the present application provides a non-transitory readable storage medium that stores computer program code, and when the computer program code is executed by a computing device, the computing device Perform the foregoing first aspect or the method provided in any one of the possible implementation manners of the first aspect. The storage medium includes but is not limited to volatile memory, such as random access memory, non-volatile memory, such as flash memory, hard disk (English: hard disk drive, abbreviation: HDD), solid state drive (English: solid state drive, Abbreviation: SSD).

In an eighth aspect, the present application provides a computer program product. The computer program product includes computer program code. When the computer program code is executed by a computing device, the computing device executes the foregoing first aspect or any of the first aspects. The methods provided in the possible implementations. The computer program product may be a software installation package. In the case where the method provided in the foregoing first aspect or any possible implementation of the first aspect needs to be used, the computer program product may be downloaded and executed on a computing device. Program product.

Description of the drawings

In order to more clearly illustrate the technical methods of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments.

Figure 1 is a schematic diagram of the blind areas of different trucks provided by an embodiment of the application;

2A is a schematic diagram of the deployment of a detection device provided by an embodiment of the application;

2B is a schematic diagram of the deployment of another detection device provided by an embodiment of the application;

2C is a schematic diagram of deployment of yet another detection device provided by an embodiment of this application;

FIG. 3 is a schematic structural diagram of a computing device 100 provided by an embodiment of this application;

4 is a schematic structural diagram of a training device 200 and a detection device 300 provided by an embodiment of the application;

FIG. 5 is a method for detecting a blind area of a vehicle according to an embodiment of the application;

Fig. 6 is a schematic diagram of a blind spot danger in a blind spot of a vehicle provided by an embodiment of the application;

FIG. 7 is a method for target detection and target tracking provided by an embodiment of the application;

FIG. 8 is a specific method for determining the blind area of a vehicle according to an embodiment of the application;

FIG. 9 is a schematic diagram of a determined blind area of a vehicle according to an embodiment of the application;

Fig. 10 is a schematic diagram of a system provided by an embodiment of the application.

Detailed ways

The solutions in the embodiments provided in this application will be described below in conjunction with the drawings in this application.

Various vehicles are often running on traffic roads, such as engineering vehicles (such as loading and unloading trucks, cement trucks, tank trucks, trucks), cars, bicycles, buses, etc. For different vehicles, due to their different models and internal design structures, the angle of view and field of view of the traffic road that the driver can observe when driving the vehicle normally on the seat is also different, although each vehicle is currently equipped with left and right mirrors and rear view mirrors. Sight glasses and other tools that assist the driver in observing the driving environment around the vehicle, but the driver still cannot see some areas around the vehicle when driving the vehicle normally, that is, blind spots. In this application, the blind zone refers to the area around the vehicle that is not observed when the driver drives the vehicle normally, and the position of the blind zone moves with the driving of the vehicle. Each vehicle has its own blind area information. The blind area information of each vehicle refers to the parameters of the blind area that the driver may not be able to observe due to factors such as the structure of the vehicle, the vehicle type, and the position of the driver's seat. The blind area information includes the blind area information. Number, shape of blind zone, position of blind zone relative to vehicle, etc. Figure 1 is a schematic diagram of the blind spots of two trucks. Different vehicles have different blind spot information. According to the blind spot information and the position information of the vehicle, the position and distribution of the blind spots of the driver during the driving of the vehicle can be determined. For construction vehicles, because the body of the construction vehicle is usually large, the driver's vision is often blocked by the vehicle body during driving, resulting in a large blind area when the driver drives the construction vehicle. When there are dynamic hazards such as other moving vehicles, pedestrians, objects, etc. in the blind area of the vehicle, or static hazards in the blind area (such as construction hazards, geographic defects, etc.), it will not only bring great danger to the vehicle and vehicle drivers and passengers , It will also bring great danger to people and things in the blind zone. In this application, situations where there are dynamic hazards and static hazards in the blind zone are collectively referred to as blind zone hazards. Detecting the blind area of the vehicle and further detecting the danger of the blind area is of great significance to the safety of vehicles and pedestrians on the traffic road.

The present application provides a method for detecting a blind area of a vehicle, and the method is executed by a detecting device. The function of the detection device for detecting the blind zone can be realized by a software system, can also be realized by a hardware device, or can be realized by a combination of a software system and a hardware device.

The deployment of the detection device is flexible, and it can be deployed in an edge environment. For example, the detection device can be an edge computing device in the edge environment or a software device running on one or more edge computing devices. The edge environment refers to the distance to be detected. A collection of data centers or edge computing devices that are close to traffic roads. The edge environment includes one or more edge computing devices. The edge computing devices may be roadside devices with computing capabilities installed on the side of the traffic road. For example, as shown in Fig. 2A, the detection device is deployed at a location close to the intersection, that is, the edge computing device on the roadside. A networkable camera is installed at the intersection. The camera captures and records the video data of vehicles passing by at the intersection and sends the video data to the detection device via the network. The detection device detects the blind area of the vehicle driving at the intersection according to the video data, and further performs the blind area danger detection. When the vehicle driving at the intersection is detected as a blind area danger, it can be through the network (for example: wireless network or car networking network) Send a blind zone warning to the on-board system in the vehicle, so that the on-board system in the vehicle reminds the driver that there is a dangerous situation in the blind zone. Or the detection device sends a blind zone alarm to the roadside warning device, so that the roadside warning device emits warning signals such as sound or light. Or, the detection device sends the detected alarm data and statistical data to the equipment or device in the traffic management system, so that the commander can conduct corresponding command or law enforcement on the vehicles on the traffic road based on the obtained alarm data and statistical data , Where the warning data includes one or more of the following information: the location and range of the blind zone on the traffic road where the blind zone danger occurs, the location information of other targets in the blind zone on the traffic road, and the types of other targets in the blind zone .

The detection device can also be deployed in a cloud environment, which is an entity that uses basic resources to provide cloud services to users in a cloud computing mode. The cloud environment includes a cloud data center and a cloud service platform. The cloud data center includes a large number of basic resources (including computing resources, storage resources, and network resources) owned by a cloud service provider. The computing resources included in the cloud data center can be a large number of computing resources. Device (for example, server). The detection device can be a server in a cloud data center that is used to detect the blind area of a moving vehicle; the detection device can also be a virtual machine created in a cloud data center to detect the blind area; the detection device can also be deployed in the cloud A software device on a server or virtual machine in a data center. The software device is used to detect the blind area of a driving vehicle. The software device can be distributed on multiple servers or distributed on multiple virtual machines On virtual machines and servers. For example, as shown in FIG. 2B, the detection device is deployed in a cloud environment, and a networkable camera set on the side of the traffic road sends the captured video data to the detection device in the cloud environment. The detection device performs blind zone detection and blind zone danger detection on the video recorded vehicle according to the video data. When a vehicle is detected as a blind zone danger, it sends a blind zone warning to the on-board system of the vehicle, so that the on-board system in the vehicle prompts driving There is a dangerous situation in the blind spot. Or the detection device sends a blind zone alarm to the roadside warning device, so that the roadside warning device emits warning signals such as sound or light. Or, the detection device sends the detected alarm data to the equipment or device in the traffic management system, so that the commander can conduct corresponding command or law enforcement on the vehicles on the traffic road based on the obtained alarm data.

The detection device can be deployed in a cloud data center by a cloud service provider. The cloud service provider abstracts the function provided by the detection device into a cloud service, and the cloud service platform allows users to consult and purchase this cloud service. After purchasing this cloud service, the user can use the service provided by the detection device of the cloud data center to detect the vehicle's blind spot danger. The detection device can also be deployed by the tenant in the computing resources (such as virtual machines) of the cloud data center rented by the tenant. The tenant purchases the computing resource cloud service provided by the cloud service provider through the cloud service platform, and runs the detection device in the purchased computing resources , So that the detection device performs blind spot detection on the vehicle.

When the detection device is a software device, the detection device can be logically divided into multiple parts, each of which has different functions (multiple parts such as: the detection device includes a target detection and tracking module, a target positioning module, a posture determination module, and a blind zone Determine the module). Several parts of the detection device can be respectively deployed in different environments or equipment, and the various parts of the detection device deployed in different environments or equipment cooperate to realize the function of vehicle blind zone hazard detection. For example: as shown in Figure 2C, the target detection and tracking module in the detection device is deployed on the edge computing device, and the target positioning module, posture determination module, and blind spot determination module are deployed in the cloud data center (for example, the server or On the virtual machine). The camera set at the traffic intersection sends the captured video data to the target detection and tracking module deployed in the edge computing device. The target detection and tracking module detects and tracks the vehicles, pedestrians and other targets recorded in the video data based on the video data , Send the obtained position information of the target in the video at the current moment, the motion trajectory information formed in the video at the current and historical moments of the target to the cloud data center, and the target positioning module and posture determination module deployed on the cloud data center , The blind spot determination module further analyzes and processes data such as the position and running track of the target in the video, and obtains the blind spot of the vehicle (and the blind spot risk determination result). It should be understood that this application does not restrict the division of the parts of the detection device, nor does it restrict the environment in which the detection device is specifically deployed. In actual applications, it can be deployed adaptively according to the computing capabilities of each computing device or specific application requirements. It is worth noting that in one embodiment, the camera can be a smart camera with certain computing capabilities, and the detection device can also be deployed in three parts, of which one is deployed in the camera, one is deployed in the edge computing device, and the other is deployed in the cloud. Computing equipment.

When the detection device is a software device, the detection device can be separately deployed on a computing device in any environment (cloud environment, edge environment, terminal computing equipment, etc.); when the detection device is a hardware device, the detection device can be in any environment Of a computing device. FIG. 3 provides a schematic structural diagram of a computing device 100. The computing device 100 shown in FIG. 3 includes a memory 101, a processor 102, a communication interface 103, and a bus 104. Among them, the memory 101, the processor 102, and the communication interface 103 implement a communication connection between each other through the bus 104.

The memory 101 may be a read only memory (Read Only Memory, ROM), a static storage device, a dynamic storage device, or a random access memory (Random Access Memory, RAM). The memory 101 may store computer instructions. When the computer instructions stored in the memory 101 are executed by the processor 102, the processor 102 and the communication interface 103 are used to execute the method for detecting the blind area of the vehicle. The memory can also store data. For example, a part of the memory 101 is used to store data required for detecting the blind area of the vehicle, and used to store intermediate data or result data during program execution.

The processor 102 may adopt a general-purpose central processing unit (Central Processing Unit, CPU), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a graphics processing unit (graphics processing unit, GPU), or any combination thereof. The processor 102 may include one or more chips, and the processor 102 may include an AI accelerator, such as a neural network processor (neural processing unit, NPU).

The communication interface 103 uses a transceiver module such as but not limited to a transceiver to implement communication between the computing device 100 and other devices or communication networks. For example, the data required to detect the danger of the blind zone of the vehicle can be obtained through the communication interface 103.

The bus 104 may include a path for transferring information between various components of the computing device 100 (for example, the memory 101, the processor 102, and the communication interface 103).

When the detection device executes the method for detecting the blind area of a vehicle provided in the embodiments of the application, it needs to adopt an artificial intelligence (AI) model. The AI model includes multiple types, and the neural network model is one of the AI models. When describing the embodiments of the present application, a neural network model is taken as an example. It should be understood that other AI models can also be used to complete the functions of the neural network model described in the embodiments of the present application, which is not limited in this application.

Neural network model is a kind of mathematical calculation model that imitates the structure and function of biological neural network (animal's central nervous system). A neural network model can include a variety of neural network layers with different functions, and each layer includes parameters and calculation formulas. According to different calculation formulas or different functions, different layers in the neural network model have different names. For example, the layer that performs convolution calculations is called convolutional layer, and the convolutional layer is often used for input signals (for example: image ) Perform feature extraction. A neural network model can also be composed of a combination of multiple existing neural network models. Neural network models with different structures can be used in different scenarios (for example: classification, recognition) or provide different effects when used in the same scenario. The different neural network model structures specifically include one or more of the following: the number of network layers in the neural network model is different, the order of each network layer is different, and the weights, parameters or calculation formulas in each network layer are different. There are many different neural network models with high accuracy for application scenarios such as recognition or classification in the industry. Among them, some neural network models can be trained by a specific training set to complete a task alone or be combined with other neural network models. A combination of network models (or other functional modules) completes a task. Some neural network models can also be directly used to complete a task alone or combined with other neural network models (or other functional modules) to complete a task.

In an embodiment of the present application, two different neural network models are required to execute the method of detecting the blind area of the vehicle. One is a neural network model used to detect targets in video data, which is called a target detection model. It should be understood that the target detection model in the embodiments of the present application can use any of the neural network models that have been used for target detection with better effects in the industry, for example: one-stage unified real-time target detection (you only look once: unified, real-time object detection, Yolo) model, single shot multi-box detector (SSD) model, regional convolutional neural network (regional convolutional neural network, RCNN) model, or fast regional convolutional neural network (fast region convolutional neural network, Fast-RCNN) model, etc.

In the embodiments of the present application, another neural network model that needs to be used to perform the method of detecting the blind area of a vehicle is: a model used to perform attribute detection on a detected vehicle, which is called a vehicle attribute detection model, and a vehicle attribute detection model is also Any one of some existing neural network models in the industry can be used, for example: convolutional neural network (convolutional neural network, CNN) model, Resnet model, Densenet model, VGGnet model, etc. It should be understood that a neural network model developed in the industry that can realize target detection and vehicle attribute detection in the future can also be used as the target detection model and vehicle attribute detection model in the embodiments of this application, which is not limited in this application.

The target detection model and the vehicle attribute detection model can be trained by the training device before being used for the blind spot detection of the vehicle. The training device uses different training sets to train the target detection model and the vehicle attribute detection model. The training device is completed The target detection model and the vehicle attribute detection model can be deployed in the target detection and tracking module in the detection device, and the detection device is used to detect the blind zone danger of the vehicle.

FIG. 4 provides a schematic structural diagram of a training device 200 and a detection device 300. The following describes the structure and function of the training device 200 and the detection device 300 with reference to FIG. 4. It should be understood that the embodiment of the present application only exemplarily divides the structure and function modules of the training device 200 and the detection device 300, and the present application does not describe them. Make any restrictions on the specific division.

The training device 200 is used to train the target detection model 203 and the vehicle attribute detection model 204 separately. Training the target detection model 203 and the vehicle attribute detection model 204 requires two training sets, which are called target detection training set and vehicle attribute detection respectively. Training set. The obtained target detection training set and vehicle attribute detection training set are stored in the database. A collection device can collect multiple training videos or training images, and the collected multiple training videos or training images are processed and annotated manually or by the collection device to form a training set. When the collection device collects multiple training videos, the collection device uses video frames in the training videos as training images, and then processes and labels the training images to construct a training set. When the training device 200 starts training the target detection model 203, the initialization module 201 first initializes the parameters of each layer in the target detection model 203 (that is, assigns an initial value to each parameter), and then the training module 202 reads The training images in the target detection training set in the database train the target detection model 203 until the loss function in the target detection model 203 converges and the loss function value is less than a certain threshold or all the training images in the target detection training set are used for training, then The target detection model 203 has been trained. Similarly, when the training device 200 starts to train the vehicle attribute detection model 204, the initialization module 201 first initializes the parameters of each layer in the vehicle attribute detection model 204 (that is, assigns an initial value to each parameter), and then The training module 202 reads the training images in the vehicle attribute detection training set in the database to train the vehicle attribute detection model 204, until the loss function in the vehicle attribute detection model 204 converges and the loss function value is less than a specific threshold or all vehicle attribute detection training sets The training image of is used for training, and the training of the vehicle attribute detection model 204 is completed.

It is worth noting that the target detection model 203 and the vehicle attribute detection model 204 can also be trained by two training devices respectively, and the target detection model 203 and/or the vehicle attribute detection model 204 may not need to be trained by the training device 200, for example: The target detection model 203 and/or the vehicle attribute detection model 204 use a neural network model that has been trained by a third party and has good accuracy for target detection and/or attribute detection.

In an embodiment of the present application, there may be no need for an acquisition device to collect training images or training videos, and no need to construct a target detection training set and/or vehicle attribute detection training set, for example: target detection training set and/or vehicle attribute detection The training set is obtained directly from a third party. In addition, it is worth noting that the training images in the target detection training set in this application can have the same content as the training images in the vehicle attribute detection training set but with different labels. For example, the collection device collects 10,000 images containing vehicles on various traffic roads. , Pedestrians, static objects and other target images. When constructing the target detection training set, the targets in the 10,000 images are labeled with bounding boxes, and the 10,000 training images after the bounding box labeling constitute the target detection training set. When constructing the vehicle attribute detection training set, the vehicles in these 10,000 images are marked with bounding boxes, and each bounding box corresponds to the attributes of the vehicle (for example: car model, vehicle brand, etc.), passing through the bounding box and These 10,000 training images after attribute labeling constitute the vehicle detection training set.

It is worth noting that, in an embodiment of the present application, the detection device may also use only one neural network model when detecting the blind area of the vehicle, which may be called a detection and recognition model, and the detection and recognition model is a target A neural network model for all functions of the detection model 203 and the vehicle attribute detection model 204. The detection and recognition model can detect the target position or identify the vehicle in the detected target, and then perform the attribute detection on the identified vehicle. The training of the detection and recognition model is the same as the training of the target detection model 203 and the vehicle attribute detection model 204, and will not be repeated here.

The target detection model 203 and the vehicle attribute detection model 204 trained by the training device 200 can be used to perform target detection and vehicle attribute detection on video frames in the video data captured by the camera, respectively. In an embodiment of the present application, as shown in FIG. 4, the trained target detection model 203 and the vehicle attribute detection model 204 are deployed to the detection device 300. In the detection device 300, the trained target detection model 203 and the training The completed vehicle attribute detection model 204 is deployed to the target detection and tracking module 301.

As shown in FIG. 4, the detection device 300 includes a target detection and tracking module 301, a target positioning module 302, a posture determination module 303, and a blind spot determination module 304.

The target detection and tracking module 301 is used to receive video data captured by the camera, and the video data captured by the camera may be a real-time video stream. The real-time video stream records the operation of the target on the traffic road at the current moment, detects the target in the video data, and obtains the position information of the target in the video at the current moment. Further, the target detection and tracking module 301 is also used for According to the location information of the target in the video at the current moment and the location information of the target in the video at the historical moment, track and fit the running trajectory of the target in the video screen over a period of time to obtain the motion of each target in the video Track information.

It should be understood that the target in this application is an entity located on a traffic road, and the target includes a dynamic target and a static target on the traffic road. The dynamic target on the traffic road is moving and running on the traffic road over time. Objects that form a running track include: vehicles, pedestrians, animals, etc.; static targets are stationary on traffic roads for a period of time, for example: static targets can be vehicles parked on the side of the road with their flames out, and construction areas formed by road construction. The target detection and tracking module 301 can identify dynamic targets and static targets during target tracking, and can track only dynamic targets, or both dynamic targets and static targets.

It should be understood that the target detection and tracking module 301 may receive video data shot by at least one camera, and detect and track the target in the video frame of the video data according to each video data.

Optionally, the target detection and tracking module 301 may also receive radar data sent by a radar device, and combine video data and radar data to jointly perform target detection and tracking.

The target positioning module 302 is configured to communicate with the target detection and tracking module 301, receive the motion track information of each target in the video data sent by the target detection and tracking module 301, and use the pre-obtained calibration relationship to place each target in the video The trajectory information in is converted into the trajectory information of each target on the traffic road.

It should be understood that the motion trajectory information of each target in the video is a sequence of pixel coordinates, which consists of the pixel coordinates of the target in different video frames. The motion trajectory information of each target in the video indicates that the target The running status in the video during a historical period of time. The trajectory information of each target on the traffic road is a geographic coordinate sequence, which is composed of the geographic coordinates of the target on the traffic road. The trajectory information of each target on the traffic road indicates the target’s current time including the current time. The operating conditions on the traffic road in a historical period. The pixel coordinates of the target are the coordinates of the pixel at the location of the target in the video frame, and the pixel coordinates are two-dimensional coordinates; the geographic coordinates of the target are the coordinates of the target in any coordinate system in the physical world, for example: this application The geographic coordinates in the use of three-dimensional coordinates composed of longitude, latitude and altitude corresponding to the location of the target in the traffic road. Optionally, the target positioning module 302 can also be used to predict the target's movement on the traffic road at a certain time or a certain period of time in the future based on the target's movement trajectory information during a period of time including the current moment and the historical moment. Position information and motion track information.

The posture determination module 303 is used to determine the posture of the target according to the trajectory information of the target on the traffic road or the result of target detection. Optionally, the attitude determination module 303 may also only determine the attitude of the vehicle.

It should be understood that in this application, the posture of the target indicates the direction of travel of the target in the physical world. For a vehicle, the heading of the vehicle or the tangent direction of the vehicle's trajectory on a traffic road can be used to indicate the posture of the vehicle. For pedestrians , The tangent direction of the trajectory of the pedestrian can be used to express the posture of the pedestrian.

The blind spot determination module 304 is used to determine the structural attributes of the vehicle (for example, the model of the vehicle, the shape of the vehicle, etc.), determine the blind spot information of the vehicle in the blind spot information database according to the structural attributes of the vehicle, and further determine the current driving according to the blind spot information of the vehicle The blind area of the vehicle (including: determining the location of the blind area on the traffic road and the distribution range of the blind area). Optionally, the blind spot determination module 304 is also used to determine whether the vehicle has a blind spot danger within the blind spot range of the location (that is, whether there are other targets in the blind zone at the current moment), and if so, send a blind spot warning to the vehicle in the blind spot danger .

Optionally, the blind spot determination module 304 may also be used to send alarm data to the traffic management system or roadside alarm equipment.

Optionally, the blind spot determination module 304 can also be used to count the blind spot risk data of the traffic road for a period of time, and send the statistical data to the traffic management system.

Due to the functions of the above-mentioned modules, the detection device provided by the embodiments of the present application can be used to detect the blind spots of vehicles in a certain geographic area at the current time or in the future in real time, and promptly warn vehicles and pedestrians in danger. Early warning can effectively reduce traffic hazards caused by blind spots of vehicles.

The method for detecting the blind area of a vehicle provided by an embodiment of the present application will be described in detail below with reference to FIG. 5.

S401: Receive video data, perform target detection and target tracking according to video frames in the video data, and obtain target type information and motion track information of the target in the video screen.

Specifically, the video data captured by a camera set at a fixed position in a traffic road is received. The video data in this application may be a video stream captured by a camera in real time. S401 acquires the video stream at the current moment in real time, detects the target in the video stream at the current moment, and detects the target's position in the video stream according to the location information and The position information of the target in the video stream at the historical moment determines the motion track information of the target in the video screen of a historical period including the current moment. The position information of the target in the video stream at the historical moment is detected by the historical moment. Obtained at the time and stored in a detection device or other readable device.

Target detection needs to adopt a target detection model, specifically: the image in the received video data is input to the target detection model, the target detection model detects the target in the image, and outputs the target in each image. Location information and type information of the target. Further, according to the position information and type information of the target in the multiple images in a continuous period of history and the target tracking algorithm, the running track of the target in the video screen is tracked to obtain the moving track information of the target in the video screen. It should be understood that the motion trajectory information of the target in the video picture represents the motion trajectory of the target in the video picture in a historical time including the current moment, and the end of the motion trajectory information (that is, the last pixel coordinate value in the pixel coordinate sequence) is The position of the target in the video frame at the current moment. The specific steps of this step S401 in an embodiment are described in detail later.

Optionally, S401 can also receive radar data sent by radar equipment (such as lidar, millimeter wave radar) installed on the traffic road, and S402 can combine target information included in the radar data when detecting and tracking targets. (For example: target position information, target contour information, etc.) and the information obtained from the detection and tracking of the target in the video frame, to obtain the position information of the target in the image, the type information, and the movement track information of the target in the video screen .

It is worth noting that in S401, it is possible to receive video data taken by multiple cameras located at different positions on the traffic road, and detect and track the target in each video data respectively, so that the method provided in this application can detect Blind areas of vehicles on traffic roads under the perspective of multiple cameras in a geographic area.

S402: Convert the movement track information of the target in the video picture into the movement track information of the target on the traffic road.

In the foregoing step S401, the motion track information of the detected multiple targets in the video screen is obtained. The motion track information of the target in the video data is the pixel coordinate sequence corresponding to the target, and the pixel coordinate sequence includes multiple pixel coordinates. , Each pixel coordinate represents the position of the target in the image of a video frame in the video screen.

Convert the target's motion trajectory information in the video screen into the target's motion trajectory information on the traffic road, in other words, convert each pixel coordinate in the pixel coordinate sequence corresponding to the target into geographic coordinates, each geographic coordinate Represents the actual position of the target on the traffic road in the physical world. The conversion of pixel coordinates into geographic coordinates requires a calibration relationship, which is the mapping relationship between the video picture taken by the camera set on the traffic road and the traffic road in the physical world, that is, the relationship between each pixel in the video picture The mapping relationship between the pixel coordinates and the geographic coordinates of the point on the traffic road in the physical world. Through the calibration relationship and the pixel coordinates in the motion track information of the target in the video image obtained in S401, the geographic coordinates of the target on the traffic road can be calculated, and the process of calculating the geographic coordinates of the target may also be called target positioning.

In an embodiment, the calibration relationship between the video image captured by each camera and the traffic road in the physical world needs to be calculated in advance. The method for calculating the calibration relationship may be:

1. Collect in advance the geographic coordinates of some control points on the traffic road that can be captured by the camera. The control point of the traffic road usually selects the sharp point of the background object in the traffic road in order to intuitively find the position of the pixel point of the control point in the video frame. For example, the right-angle points of traffic signs, sharp points of arrows, corners of green belts, etc. in traffic roads are used as control points. The geographic coordinates (longitude, latitude, altitude) of the control points can be collected manually or by unmanned For driving car collection, the selected control points of the traffic road need to be evenly distributed on the traffic road, and the number of control points selected needs to refer to the actual situation.

2. Obtain the pixel coordinates of the captured control point in the video frame captured by the camera. Specifically, read the video of the traffic road taken by the camera fixedly set on the traffic road, and obtain the pixel coordinate value corresponding to the control point in any video frame taken by the camera, which can be obtained manually, that is, manually observe the control point in the video frame The corresponding pixel, and record the geographic coordinates of the pixel. Obtaining the pixel coordinate value corresponding to the control point can also be obtained through the program, for example, using corner detection, short-time Fourier transform edge extraction algorithm and sub-pixel coordinate fitting method to obtain the corresponding pixel of the control point of the traffic road in the video frame coordinate.

3. Establish the mapping relationship between the video picture under the camera's perspective and the traffic road in the physical world according to the geographic coordinates and pixel coordinates of the control point. For example, the homography transformation matrix H converted from pixel coordinates to geographic coordinates can be calculated according to the principle of homography transformation. The homography transformation formula is H=(m0,n0,h0)/(x0,y0), where (m0,n0 ,h0) is used to represent the geographic coordinates of each control point, (x0, y0) is used to represent the pixel coordinates of each control point. The H matrix corresponding to the video data captured by the camera can be calculated according to the pixel coordinates (x0, y0) and geographic coordinates (m0, n0, h0) of at least three control points. It should be understood that the H matrix corresponding to the video data captured by each camera set at a different position of the traffic road is different.

The calibration matrix obtained through the above steps 1-3 is the calibration relationship between the video picture taken by the camera and the traffic road. Through the calibration matrix H and the movement track information of the target in the video frame obtained by S401 (that is, the pixel coordinate sequence of the target in different video frames), the geographic coordinates corresponding to each pixel coordinate of the target in different video frames can be obtained. The specific calculation method is: (m,n,h)=H*(x,y), where (m,n,h) is the geographic coordinates of the target to be calculated, and (x,y) is the target in the video screen The pixel coordinates in the motion trajectory information. It can be seen from the above that each pixel coordinate of the target corresponds to a geographic coordinate. After calculating the geographic coordinate corresponding to each pixel coordinate in the motion track information of the target in the video screen, the geographic coordinate sequence of the target can be obtained. It should be understood that the position of each geographic coordinate in the geographic coordinate sequence of the target in the geographic coordinate sequence is the same as the position of its corresponding pixel coordinate in the pixel coordinate sequence.

It should be understood that the movement trajectory information of the target on the traffic road indicates the movement trajectory of the target on the traffic road in a period of time including the current moment, and the end (ie the last geographical coordinate value in the geographical coordinate sequence) in the movement trajectory information is this The position of the target on the traffic road at the current moment.

It should be understood that this step S402 converts the motion trajectory information of each target in the video image obtained in the foregoing S401 to obtain the motion trajectory information of each target on the traffic road.

Optionally, after obtaining the target's motion trajectory information on the traffic road, curve fitting can be performed on the target's motion trajectory on the traffic road according to the target's motion trajectory information and function on the traffic road. Specifically, according to the motion The distribution of each coordinate point in the geographic coordinate sequence in the trajectory information and the time information corresponding to each coordinate point, select an appropriate function, use the coordinate point corresponding to each moment in the geographic coordinate sequence to calculate the parameters of the function, and finally obtain the pseudo The fitting function is a function between time information and geographic coordinate information. The geographic location of the target at the future time can be predicted according to the obtained fitting function, that is, the location information of the target on the traffic road at the future time can be obtained.

It is worth noting that in another embodiment, in order to comprehensively perform blind spot detection and blind spot risk judgment on vehicles on the traffic road, the foregoing step S401 may separately obtain video data taken by multiple cameras set on the traffic road ( For example: For a traffic intersection in four directions, you can obtain the video data shot by cameras set in the four directions respectively), perform target detection and target tracking as described in S401 on each video data, and obtain the Position information, type information of multiple targets in the image, and movement track information of the target in the video screen. Further, the method of executing S402 converts the movement trajectory information of each target in the video picture into the movement trajectory information of each target on the traffic road (need to calculate in advance the video picture taken by each camera and the calibration before the traffic road Further, the target whose motion trajectory information on the traffic road overlaps (or is similar) is determined to be the same target on the traffic road, that is, the target is captured by different cameras, and the same target corresponds to multiple targets. The movement trajectory is fused, for example, the average value of each geographic coordinate in multiple geographic coordinate sequences is taken, and each average value forms a new geographic coordinate sequence, and the new geographic coordinate sequence is determined as the target on the traffic road. Motion track information. Continue to execute S403-S405 for each target to complete the vehicle blind spot determination and blind spot risk judgment.

The method of blind spot detection and blind spot hazard determination of vehicles based on video data captured by multiple cameras can avoid the problem of limited field of view on a traffic road captured by a single camera, and can further avoid some targets in the video screen captured by a single camera It is occluded, resulting in the problem that the occluded target cannot be detected in the blind area. Further, the video data captured by multiple cameras can also be used to obtain the complete movement trajectory of the target on the traffic road within a period of time, so that the detection device can judge the posture of the vehicle according to the complete movement trajectory of the target in the subsequent steps.

S403: Determine the posture of the target according to the trajectory information of the target on the traffic road or the result of target detection.

In this application, the posture of the target is the traveling direction of the target on the traffic road. There are many methods for determining the posture of the target, and different methods can be used to determine the posture of different targets. For example, for a vehicle, a method for determining the posture of the vehicle is to use the trajectory information (ie geographic coordinate sequence) of the vehicle on the traffic road obtained in S402 to perform trajectory fitting, and use the tangent direction of the trajectory as For the posture of the vehicle, it should be understood that the tangent direction is the tangent direction of the end of the trajectory (because the trajectory obtained by fitting has a time sequence, the point on the trajectory corresponding to the current moment can be called the end). For other targets, such as pedestrians, this method can also be used to determine the posture and obtain the posture information of the target.

For vehicles, another method can be used to determine the posture of the vehicle, that is, the three-dimensional detection method is used to detect the three-dimensional contour of the vehicle in the video, and the contour of the vehicle is converted into the contour information of the vehicle on the traffic road using the aforementioned method of S402. Determine the posture of the vehicle based on the contour information and the direction of the entrance and exit lanes on the traffic road. For example: at an intersection of a traffic road, the direction of the long side of the vehicle in the contour information of a car is northwest-southeast, and the traffic direction of the entrance and exit lane corresponding to the position of the vehicle at the intersection is determined by southeast To the northwest, the posture of the vehicle is determined to be the northwest direction according to the direction of the long side of the vehicle and the direction of the entrance and exit lanes on the traffic road. It is worth noting that information such as the direction of the entrance and exit lanes on the traffic road can be obtained by the detection device from other devices in advance.

It is worth noting that for vehicles, the above two methods can also be combined to determine the posture of the vehicle.

It is also worth noting that if the above S402 obtains the fitting function of the movement trajectory of the vehicle and other targets, and the position information of the target on the traffic road in the future, the vehicle's movement trajectory curve in the future can be obtained according to the fitting function. , The tangent direction of the curve can be determined according to the trajectory curve for a period of time in the future as the posture of the vehicle at the future time, and the posture information of the vehicle at the future time can be obtained.

S404: Determine the blind area of the vehicle.

From the foregoing steps S401-S403, the type of the target on the traffic road, the position information of the target on the image and the traffic road, and the posture information of the target are obtained. According to one or more of these information, the blind zone estimation of the vehicle can be performed, and the blind zone position information and blind zone range information of the vehicle at a certain position at the current time or in the future can be obtained.

Blind zone estimation of a vehicle mainly includes: detecting vehicle attributes to determine the structural attributes of the vehicle; searching the blind zone information database according to the structural attributes of the vehicle to determine the blind zone information of the vehicle; according to the blind zone information of the vehicle, the current time or the future time The location information and posture information of the vehicle on the traffic road determine the blind area of the vehicle on the traffic road at the current time or in the future, and obtain the blind area distribution information and the blind area location information of the vehicle.

It should be understood that the blind zone information database is a pre-built database that stores the structural attributes of various vehicles and the blind zone information corresponding to the vehicles of each structural attribute. The blind zone information database can be constructed by manually collecting data in advance, or it can be purchased from a third party.

It should also be understood that the blind zone information database may be a database deployed in the detection device, or may be a database outside the detection device and capable of data communication with the detection device. In the embodiment of the present application, description is made by taking the blind area information database as a database in the detection device as an example.

The specific steps of the method for estimating the blind zone of the vehicle are described in detail in S4041-S4044 below.

Optionally, after determining the blind area of the vehicle on the traffic road, the structural attributes of the vehicle, the position information of the vehicle on the traffic road, the position and distribution of the blind area of the vehicle on the traffic road can be combined with the blind area of the vehicle on the traffic road. Construct a visual blind spot image, for example, you can obtain a real-world map, determine the model of the vehicle according to the structural attributes of the vehicle, map the model of the vehicle to the corresponding position of the real-world map according to the position information of the vehicle on the traffic road, and according to the vehicle's position on the traffic road The location and distribution of the blind area map the blind area of the vehicle to the corresponding position on the real map. The obtained visual blind spot image may be a graphical user interface (GUI), and the GUI may be sent to other display devices, for example, sent to the vehicle-mounted display device of the corresponding vehicle, or sent to the display device in the traffic management system. Through the visualized blind spot images of the vehicle at the current moment or in the future, it can be intuitively and quickly determined whether the vehicle is in a dangerous state, and the driving direction can be adjusted in time to avoid the danger of blind spots.

S405: Perform a blind zone risk judgment on the blind zone of the vehicle, and send a blind zone warning to the vehicle or other vehicles or people in the blind zone when the blind zone is dangerous.

After the blind zone of the vehicle at the current time or the future time is determined by the foregoing step S404, the position of other targets on the traffic road or the movement trajectory information of other targets on the traffic road obtained in the foregoing steps S401-S402 can be used to determine the current moment Or whether the position of other targets in the future is in the blind zone of the vehicle, or whether there are other targets in the blind zone can be detected according to the position and range of the blind zone at the current time or in the future. For example, as shown in Figure 6, at the current moment, if there is a target in the blind area of the vehicle, it is considered that the vehicle is in danger of the blind area, and the vehicle is further sent to the in-vehicle system through the network (for example: wireless network or car networking network) Blind zone warning. The blind zone warning includes warning data, which can include one or more of the following: the position of the blind zone where the blind zone danger occurs, the position of the target in the blind zone, the type of the target in the blind zone, etc., so that the vehicle in the vehicle The system reminds the driver that there is a dangerous situation in the blind spot.

Optionally, when detecting that there are other targets in the blind area of the vehicle, the detection device may send an alarm to the roadside warning device, so that the roadside warning device emits warning signals such as sound or light.

Optionally, the detection device can also send the detected alarm data to the equipment or device in the traffic management system, so that the commander can conduct corresponding command or law enforcement on the vehicles on the traffic road based on the obtained alarm data.

Optionally, the detection device can also record the blind zone risk data of each vehicle at historical moments. The blind zone risk data can include: the vehicle model where the blind zone danger occurs, the time of occurrence, the position of the blind zone where the blind zone danger occurs, and the target information in the blind zone. The detection device can also collect statistics on the blind zone risk data of the vehicle at the current moment and the blind zone risk data at the historical moment, obtain statistical data, and send the statistical data to the traffic management platform, so that the traffic management platform can perform statistics on the traffic area. Risk assessment or adaptive planning and management based on statistical data or the introduction of corresponding regulatory strategies.

Optionally, according to the location and type of other targets in the blind zone detected in S405, and the blind zone of the vehicle on the traffic road at the current or future moments, the visual blind zone image can be further constructed based on the previously constructed visual blind zone image. The model corresponding to other targets in the blind zone is projected to the corresponding position in the visual blind zone image, so that the obtained visual blind zone image can intuitively reflect that the vehicle is in danger of the blind zone, and can indicate that the blind zone danger is the current or future time of the vehicle. There is a certain type of target in a certain position in a certain blind zone. This allows the driver to quickly understand the current dangerous situation after obtaining the current visualized blind spot image, and make timely avoidance strategies.

Optionally, after the detection device determines the risk of the blind zone of the vehicle in S405, it can also perform a renewal for the vehicle based on the obtained current position information of other targets in the blind zone and the position information of other targets in the future, as well as the current driving route of the vehicle. Carry out route planning to avoid conflicts between vehicles and vehicles in the blind area, causing traffic accidents. After the detection device performs route planning, it can generate an adjustment instruction. The adjustment instruction can include the new driving route information planned by the detection device. The detection device further sends the adjustment instruction to the vehicle, so that the vehicle can promptly respond to the vehicle after receiving the adjustment instruction. The driving route is adjusted, for example, after the autonomous vehicle receives the adjustment instruction, it will continue to drive according to the new driving route in the adjustment instruction to realize the blind spot danger avoidance. Or the detection device sends the adjustment instruction to the traffic management platform, and the traffic management platform commands the vehicles. Optionally, the detection device can also plan a route for other targets in the blind zone, and generate adjustment instructions according to the planned new route information. The adjustment instructions include the new route information and send the adjustment instructions to other targets in the blind zone. (For example: self-driving vehicles), so that other targets can adjust future routes according to adjustment instructions, which can also eliminate the danger of blind spots.

Optionally, when the blind zone risk determination is performed on the blind zone of the vehicle, if the obtained blind zone information of the vehicle includes the risk coefficient of the blind zone, the blind zone with a higher risk coefficient can be selected according to the risk factor of the blind zone in the blind zone information of the vehicle As a high-risk blind zone, use the blind zone information of the high-risk blind zone to make a blind zone hazard judgment on the high-risk blind zone. For example, if a blind zone with a blind zone risk factor greater than a preset danger threshold is determined as a high-risk blind zone, then when making a blind zone risk judgment It only needs to judge whether there are other targets in the high-risk blind zone with a high risk factor.

Optionally, when the blind zone risk determination is performed on the blind zone of the vehicle, if the obtained blind zone information of the vehicle includes the risk factor of the blind zone, the risk factor of the blind zone can be corrected according to the position and range of the blind zone on the traffic road, so that The revised risk factor of the blind zone can more accurately reflect the risk of the vehicle in the blind zone of the traffic road; according to the relationship between the revised risk factor of the blind zone and the preset risk threshold, for example, the revised The blind zone with the blind zone risk coefficient greater than the preset risk threshold is determined as the high-risk blind zone of the vehicle on the traffic road, and then the blind zone risk judgment is performed.

It is worth noting that when the blind zone hazard determination is performed on the blind zone of the vehicle, if only the position and range of the high-risk blind zone of the vehicle on the traffic road are obtained in S404, it is only necessary to check whether the high-risk blind zone of the vehicle is dangerous. Make judgments.

The above-mentioned method of only judging the danger of the blind zone in the high-risk blind zone with a higher risk factor can save the time for the detection device to judge the danger of the blind zone, because some blind zones with a lower risk factor are less likely to have other targets, and on the other hand On the one hand, it can also make the blind zone warning data sent by the detection device more accurate, because even if there are other targets in the blind zone with a small risk factor of the vehicle, the possibility of collision or friction between the vehicle and the target in the blind zone during driving is also possible. Very small, therefore, for the presence of other targets in these blind spots, it is not considered a blind spot hazard, and there is no need to send warning data to vehicles or traffic management systems.

Through the above steps S401-S405, blind spot detection and blind spot hazard judgment can be realized for the vehicles running on the traffic road, so that the driver driving the vehicle can learn about the blind spot danger in time and carry out timely danger avoidance. Through the above steps, the traffic management system can also alert targets (such as pedestrians, non-motor vehicles, etc.) in the blind area of the vehicle based on the warning data, such as: horns, flashlights, and buzzers through roadside warning equipment Alarms, etc.

It is worth noting that the aforementioned steps S401-S403 of this application can be performed on all targets on the traffic road, and the aforementioned steps S404-S405 can be performed on all motor vehicles on the traffic road, so that all vehicles can learn about the dangerous situation of the blind zone in time. The aforementioned steps S404-S405 can also be performed for a specific type of vehicle on a traffic road, for example: blind spot detection and blind spot risk judgment are only performed on construction vehicles with larger vehicles, because this type of vehicle is more likely to have blind spot dangerous events . It is possible to detect the type of vehicle for blind spot detection and blind spot hazard detection during target detection in the aforementioned step S401, then determine the type of vehicle to be detected before executing S404, and execute the subsequent steps of S404 and S405 only for the type of vehicle to be detected .

It is also worth noting that the present application can perform the method of steps S401-S405 on the video frame (or the video frame at a fixed time interval) captured by the camera at each moment, that is, the detection device of the present application can realize continuous and real-time vehicle detection. The blind spot detection and blind spot risk determination.

The specific implementation method of step S401 is described below in detail with reference to FIG. 7:

S4011: Receive video data, extract images (that is, video frames) in the video data, and standardize the size of the images. It should be understood that the video data is a video stream on a traffic road captured by a camera in real time, and processing the image in the video data can be understood as processing the picture on the traffic road at the current moment.

The purpose of standardizing the size of the image in this step is to make the size of the standardized image adaptable to the input of the target detection model.

This application does not specifically limit the method for standardizing the size of the image. For example, a method of stretching or compressing the size or a method of filling or cropping can be used.

S4012: Input the standardized image to the target detection model, and obtain the position information and type information of the target in the image. Specifically, the target detection model performs feature extraction on the input image, and further detects the target in the image based on the extracted features. The target detection model outputs the location information and type information of the target in the detected image, for example: target detection The model outputs an output image, the detected target in the output image is framed by a rectangular frame, and each rectangular frame also includes the type information of the target. The position information of the target in the image is the pixel coordinates of one or more points in the image, for example: it can be the pixel coordinates of the rectangular frame corresponding to the target, or the center or the lower left corner of the rectangular frame corresponding to the target Pixel coordinates. The type information of the target is the type to which the target belongs, for example: pedestrians, vehicles, or static objects.

It should be understood that the reason why the target detection model can perform target detection on the input image is because the target detection model is trained by the target detection training set before the target detection is performed. For example, in the embodiment of the present application, the target detection model must be To detect traffic, vehicles, and static objects on traffic roads, it is necessary to use multiple images of pedestrians, vehicles, and static objects in the target detection training set to train the target detection model, and each image in the target detection training set is Annotated, specifically, pedestrians, vehicles or static objects contained in each image are framed by rectangular boxes, and each rectangular box corresponds to the type information of the target in the rectangular box. Because the target detection model repeatedly learns the characteristics of each type of target during the training process, the trained target detection model has the ability to detect pedestrians, vehicles, and static objects in the input image.

It is worth noting that S4012 performs target detection on the images in the continuous time video data received in S4011 (or images at a fixed time interval). Therefore, after S4012 is executed for a period of time, multiple continuous times can be obtained. The location information and type information of the target in the image in the video data. And each image corresponds to a time stamp, and the images can be sorted in time order through the time stamp.

S4013: Track the moving target according to the position information and type information of the target in the image detected in S4012, and determine the moving track of the target in the video frame in the historical time period including the current moment.

Target tracking refers to tracking the targets in two images (or two images at a fixed time interval) in the video data at adjacent moments, to determine that the targets in the two adjacent images are the same target in the physical world, so that the two Two targets in each image correspond to the same target ID, and the pixel coordinates of the target ID in the image at the current moment are recorded on the target trajectory table. The target trajectory table is recorded in each of the areas captured by the camera. The pixel coordinates of the target at the current and historical moments (the trajectory of the target can be fitted by the pixel coordinates of the target's current and historical moments). During target tracking, the type information and location information of the target obtained in step S4012 are compared with the type information and location information of the target in the buffered and processed video frame at the previous time to determine the adjacent time (or fixed time interval). The association between the targets in the two images at the two moments), that is, the targets judged to be the same target in the two images are marked with the same target ID, and the target ID corresponding to each target and its image A target ID corresponds to the pixel coordinates in. Thus, after target tracking is performed on the images of multiple target detections according to the time sequence, multiple pixel coordinates corresponding to a target ID can be obtained to form a pixel coordinate sequence, and the pixel coordinate sequence is the motion track information of the target.

The following provides a specific target tracking process:

1: Target match. According to the location information of the detected target in the image at the current moment (that is, the pixel coordinates of the target in the image) and type information, the detected target in the image at the current moment is compared with the image at the previous moment (or before a fixed time interval). The target in the current image) is matched, for example, the target ID of the target in the current image is determined according to the overlap ratio between the rectangular frame of the target in the current image and the rectangular frame of the target in the previous image. If the overlap ratio between the rectangular frame of the target and the rectangular frame of a target in the image at the previous moment is greater than the preset threshold, it is determined that the target at the current moment and the target at the previous moment are the same, and the target track table is found The recorded target ID corresponding to the target, and the pixel coordinates of the target in the current image are recorded in the sequence corresponding to the target ID. It should be understood that step 1 and subsequent steps are performed for each target detected in the current image.

2: When one or more targets in the current image in step 1 above do not match the targets in the image at the previous moment (that is, the one or more targets are not found in the image at the previous moment, for example: a vehicle At the current moment, you have just entered the area of the traffic intersection captured by the camera), the one or more targets are determined to be new targets on the traffic road at the current moment, and a new target ID is set for the target, where the target ID is uniquely Identify the target, record the target ID and its pixel coordinates at the current moment in the target track table.

3: When one or more targets at the previous moment in the preceding step 1 do not match the target in the image at the current moment (that is, there is a target at the previous moment, but it is not found at the current moment, for example: the target at the current moment is If another target is partially or completely occluded or the target has left the area of the traffic road captured by the camera at the current moment), the target's pixel coordinates at the historical moment recorded in the target track table are used to predict that the target is in the current moment's image Pixel coordinates (for example, using three-point extrapolation, trajectory fitting algorithm, etc.).

4. Determine the existence state of the target according to the pixel coordinates of the target predicted in step 3. When the predicted target's pixel coordinates are outside or at the edge of the current image, it can be determined that the predicted target has left the camera’s view angle at the current moment; when the predicted target’s pixel coordinates are inside the current video frame and not at the edge Position, confirm that the target is still in the video frame at the current moment.

5. When it is determined in step 4 that the predicted target has left the view of the camera at the current moment, delete the target ID and its corresponding data in the target track table.

6. When it is determined in step 4 that the predicted target is still in the image at the current moment, record the pixel coordinates of the predicted target in the target track table.

It is worth noting that the aforementioned steps 1-5 can be performed for each target detected in the image at each moment in the video data captured by the camera (or the image at a fixed time interval), or only the video captured by the camera In the image at each moment in the data (or the image at a fixed time interval), the target detection result obtained in the foregoing S4012 is non-static, and the foregoing steps 1-5 are executed.

After the target is tracked, the pixel coordinate sequence corresponding to multiple target IDs can be obtained, and the pixel coordinate sequence is the motion track information of the target in the video screen. If the target tracking operation is also performed on a static target, a pixel coordinate sequence of the static target can be obtained. Since the target is static, the pixel coordinates in the pixel coordinate sequence of the target will be gathered near a point.

The specific implementation steps of step S404 in an embodiment are described below in detail with reference to FIG. 8:

S4041: Detect the structural attributes of the vehicle.

Since the foregoing step S401 detects the target recorded in the video data captured by the camera set on the traffic road, the target includes vehicles, pedestrians, and so on.

In this step, there are two ways to detect the structural attributes of the vehicle according to the vehicle attribute detection model. One is: input each frame of image or fixed time interval images in the video data into the trained vehicle attribute detection model. The vehicle attribute detection model detects the structural attributes of the vehicles in the input image, and obtains the position information of each vehicle in the image and the structural attributes of each vehicle.

The other is: this step divides the rectangular frame corresponding to the vehicle in the image according to the position information of the target whose target type is vehicle detected in the foregoing step S401, and inputs the divided vehicle sub-image into the vehicle attribute detection model. The structural attributes of each vehicle are obtained from the vehicle attribute detection model. According to the combination of the target detection model and the vehicle attribute detection model, the position information of the target in the image, the target type information and the structural attributes of the vehicle can be obtained.

Which method is used to obtain the structural attributes of the vehicle can be determined according to the type of input image required by the trained vehicle attribute detection model.

It is worth noting that, from the foregoing, the vehicle attribute detection model is a neural network model. The structural attributes of the vehicle monitored by the vehicle attribute detection model can be the type of vehicle, such as the model of the vehicle or the sub-category of the vehicle. It is related to the function of the vehicle attribute detection model after being trained. For example: when the vehicle attribute detection model is trained, the training images used include vehicles, and the vehicles in the training images are marked with the vehicle model (for example: Changan 20-ton truck, BYD 30-seater passenger car, Mercedes-Benz c200 sedan, etc.) , The trained vehicle attribute detection model can be used to detect the model of the vehicle; another example: the training image used when the vehicle attribute detection model is trained contains the vehicle, and the vehicle in the training image is marked with the sub-category of the vehicle (such as : 7-seater commercial vehicle, 4-seater car, 20-ton truck, 10-ton cement truck, etc.), the vehicle attribute detection model that has been trained can be used to detect sub-categories of vehicles.

The structural attributes of the vehicle can also include: the length and width information of the vehicle, the type and location of the vehicle cab, etc. For step S4041, the vehicle attribute detection model can also be used to detect the length and width information of the vehicle, the type and location of the vehicle cab, etc. information. It should be understood that by detecting the structural attributes of the vehicle, one or more of structural attributes such as the model of the vehicle, the sub-category of the vehicle, the aspect ratio of the vehicle, the type and location of the cab of the vehicle, etc. can be obtained.

It is worth noting that the above step S4041 can also be performed after the target detection in S401 is performed. The obtained vehicle structural attributes and the target type and target location information obtained in the target detection stage can be stored together in the detection device. Target detection and tracking Module, or stored in other modules of the detection device, or stored in other storage devices readable by the detection device.

S4042: Query the blind area information database according to the structural attributes of the vehicle to obtain the blind area information of the vehicle.

This application does not limit the structure and form of the blind spot information database. For example, in an embodiment, the blind spot information database may be a relational database, and the blind spot information database provides a query interface. The structural attributes of the vehicle obtained by the foregoing steps can be It is sent to the interface, and the corresponding blind zone information in the blind zone information database is queried by the interface according to the structural attributes of the vehicle, and the blind zone information database returns the query result through the interface.

The query result is the blind zone information of the vehicle corresponding to the vehicle structure attribute, which can include: the position of the blind zone relative to the vehicle, the shape of the blind zone, and the number of blind zones. Among them, the position of the blind zone relative to the vehicle can be the relative key point in the blind zone. The offset from the center point of the vehicle, for example, the position of a rectangular blind area relative to the vehicle is the offset length and direction of the four corner points of the rectangular blind area relative to the center point of the vehicle. Optionally, the query result may also include: the area of the blind zone and the risk factor of the blind zone, where the risk factor of the blind zone is the probability that a dangerous situation may occur in the blind zone, and is used to indicate the degree of danger of the blind zone. The risk factor of the blind zone can be obtained from the blind zone information The library comprehensively judges the area, location, shape and other factors of the blind area of the vehicle. For example, the blind area of the vehicle that has a large blind area and is located diagonally behind the vehicle can be judged as a higher risk factor for the blind area.

There are many specific ways to query the blind zone information database according to the structural attributes of the vehicle to obtain the query results. There are three ways as examples:

1. The blind spot determination module of the detection device sends the model of the vehicle or the sub-category of the vehicle to the interface of the blind spot information database, and the blind spot information database queries the blind spot information corresponding to the model or the sub-category of the vehicle according to the model or sub-category of the vehicle. The blind spot information is returned to the interface as the query result. The query result can include: the position of the blind spot of the vehicle relative to the vehicle, the shape of the blind spot, and the number of blind spots; optionally, the query result can also include: the area of the blind spot and the risk factor of the blind spot .

2. The blind spot determination module of the detection device sends the model of the vehicle or the sub-category of the vehicle to the interface of the blind zone information database. The blind spot information database does not find the vehicle corresponding to the model or the sub-category according to the vehicle model or sub-category. Blind zone information, the blind zone information database returns a query failure message to the blind zone determination module. The blind zone information database further sends the length and width information of the vehicle, the type and location of the cab of the vehicle and other structural attributes to the blind zone information database interface. The blind zone information database is based on the length of the vehicle. Query the blind zone information of the vehicle closest to the structural attributes of the vehicle in the blind zone information database for structural attributes such as wide information, the type and location of the vehicle’s cab, as the query result, and return the query result to the interface. The query result can include: The position of the blind zone relative to the vehicle, the shape of the blind zone, and the number of blind zones; optionally, the query result can also include: the area of the blind zone and the risk factor of the blind zone.

3. The blind zone determination module of the detection device sends the structural attributes of the vehicle to the interface of the blind zone information database. The structural attributes of the vehicle include: vehicle model, vehicle sub-category, vehicle length and width information, vehicle cab type and location, etc. One or more of. The blind spot information database determines the blind spot information of the vehicle of the model or sub-category according to the structural attributes, or the blind spot information database determines the blind spot information of the vehicle closest to the structural attributes of the vehicle according to the structural attributes. The blind spot information database finally returns the query result to the blind spot determination module. The query result can include: the number of blind spots, the position of the blind spot relative to the vehicle, and the shape of the blind spot; optionally, the query result can also include: the risk factor of the blind spot, the blind spot The area.

The blind area information of the vehicle is obtained from the aforementioned S4042. The blind area information is related to the attributes of the vehicle, and vehicles with different attributes correspond to different blind area information.

S4043: Determine the blind zone of the vehicle according to the blind zone information.

In this step, the blind spot information of the vehicle obtained in step S4042, the trajectory information of the vehicle on the traffic road obtained in step S402, and the posture information of the vehicle obtained in step S403 are used to determine the blind spot of the vehicle at the current moment. Alternatively, the position information of the vehicle at the future time and the posture information of the vehicle at the future time, and the blind area information of the vehicle are obtained from the foregoing steps S401-S403, and the blind area of the vehicle at the future time can be determined.

The specific method for determining the blind zone of the vehicle at the current moment is as follows: according to the vehicle's trajectory information on the traffic road, the geographic coordinates and the posture of the vehicle at the current moment are combined with the blind zone information such as the position, area, and shape of the blind zone relative to the vehicle. The location, distribution and area of the blind area of the vehicle on the traffic road. For example: as shown in Figure 9, for a 20-ton truck of Changan, the geographic coordinates of the vehicle on the traffic road are known, that is, the geographic coordinates of the center point of the vehicle are known, and the The attitude of the car is from east to west. The blind spot information of the vehicle obtained from the aforementioned S4042 shows that the vehicle has 6 independent blind areas, the shape and area of each independent blind area, and the key points in each independent blind area relative to the vehicle The offset of the special point. Therefore, the actual position and range of the blind area of the vehicle on the traffic road can be determined according to the blind spot information and the position and posture of the vehicle on the traffic road.

Optionally, when determining the blind zone of the vehicle according to the blind zone information in step S4043, if the blind zone information of the vehicle includes the risk factor of the blind zone, in another embodiment of the present application, S4043 determines the risk according to the risk factor of the blind zone in the blind zone information The high-risk blind area information corresponding to the high-risk blind block whose coefficient is greater than the preset risk threshold is determined based on the high-risk blind area information, the current position information and posture information of the vehicle on the traffic road. The location and extent of high-risk blind spots. This method only determines the location and distribution range of the high-risk blind area of the vehicle on the traffic road, so that when the blind area is judged later, only the high-risk blind area is judged whether there is a blind area hazard, which reduces the amount of calculation and due to the low level of the vehicle. The risk of the blind zone is less likely to be dangerous. Even if there are other targets in the blind zone, the vehicle will not easily collide and rub against other targets in the low-risk blind zone when driving. Therefore, the situation where there are other targets in the low-risk blind zone is not correct. The warning will not cause danger and injury to the vehicle and other targets, and avoid the interference of the vehicle driver and pedestrians caused by excessive warning.

Optionally, when determining the blind zone of the vehicle according to the blind zone information in step S4043, the driving speed of the vehicle can also be determined according to the vehicle's movement track information on the traffic road, and the range of all or part of the blind zone of the vehicle is expanded according to the driving speed of the vehicle . For example, if the driving speed of the vehicle is higher than a certain threshold, the front blind zone range of the vehicle is multiplied by the preset proportional coefficient, so that the blind zone range is expanded. For another example, the driving speed of the vehicle is converted into a proportional coefficient according to a preset rule, the area of the blind area in the blind area information is multiplied by the proportional coefficient, and the position of the blind area in the blind area information relative to the vehicle. Specifically, the method of determining the driving speed of the vehicle according to the movement trajectory information of the vehicle on the traffic road is as follows: the distance difference between the adjacent geographic coordinates in the geographic coordinate sequence of the movement trajectory information corresponds to the adjacent geographic coordinate value The time difference between two adjacent video frames in the video data is divided to obtain the driving speed of the vehicle at a moment.

Optionally, if there is no risk factor of the blind spot in the obtained blind spot information, after determining the blind spot of the vehicle according to the blind spot information in step S4043, the risk factor of the blind spot may be determined according to the geographic location and range of each blind spot determined. Or if the blind area information obtained in the foregoing step S4042 has a risk coefficient of each blind area, the risk coefficient of the blind area is adjusted according to the geographic location and range of each blind area to make the risk coefficient of the blind area more accurate.

After the above steps S4041-S4043, the blind area of the vehicle on the traffic road at a certain moment can be determined. It should be understood that the video data captured by the camera in this application may be a video stream that records the movement of objects on the traffic road in real time. Therefore, the method of determining the blind area of the vehicle can be continuously executed to determine the blind area of the vehicle at each moment.

The present application also provides a detection device 300 as shown in FIG. 4, and the modules and functions included in the detection device 300 are as described above, which will not be repeated here.

In an embodiment, the target detection and tracking module 301 in the detection device 300 is used to execute the aforementioned method step S401, and in another more specific embodiment, the target detection and tracking module 301 is used to execute the aforementioned method step S4011. -S4013 and its optional steps; the target positioning module 302 is used to perform the aforementioned method step S402; the posture determination module 303 is used to perform the aforementioned method step S403; the blind spot determination module 304 is used to perform the aforementioned method steps S404-S405, in another In a more specific embodiment, the blind spot determination module 304 is configured to execute the aforementioned method steps S4041-S4043, S405, and the aforementioned optional steps described in S404-S405.

The present application also provides a detection system for detecting the blind area of a vehicle. The system includes a vehicle dynamic monitoring system and a vehicle blind area detection system. The vehicle dynamic monitoring system is used to receive video data, and determine the position information and posture information of the vehicle on the traffic road at the current time or in the future according to the video data, wherein the video data is captured by a camera set on the traffic road. The vehicle blind spot detection system is used to obtain the blind spot information determined by the structural attributes of the vehicle, and determine the blind spot of the vehicle on the traffic road at the current time or in the future according to the blind spot information of the vehicle, the position information and posture information of the vehicle on the traffic road. More specifically, the aforementioned detection system is used to execute the aforementioned S401-S405 methods, the aforementioned vehicle dynamic monitoring system of the aforementioned detection system is used to execute aforementioned S401-S403, and the vehicle blind spot detection system is used to execute aforementioned S404-S405.

The present application also provides a vehicle-mounted device, which is installed on a vehicle. The vehicle-mounted device of the present application can be used to execute the aforementioned methods S401-S405, and the vehicle-mounted device can provide the same functions as the detection device 300.

This application also provides a vehicle. The vehicle includes a storage unit and a processing unit. The storage unit of the vehicle is used to store a set of computer instructions and data sets. The processing unit executes the computer instructions stored in the storage unit, and the processing unit reads the The data collection of the storage unit, so that the vehicle can execute the aforementioned methods S401-S405.

The storage unit of the aforementioned vehicle may be a read only memory (Read Only Memory, ROM), a static storage device, a dynamic storage device or a random access memory (Random Access Memory, RAM). The processing unit of the vehicle may be a general-purpose central processing unit (Central Processing Unit, CPU), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a graphics processing unit (graphics processing unit, GPU), or any combination thereof. The processing unit may also include one or more chips, and the processing unit may also include an AI accelerator, such as a neural network processor (neural processing unit, NPU).

The present application also provides a computing device 100 as shown in FIG. 3. The processor 102 in the computing device 100 reads a set of computer instructions stored in the memory 101 to execute the aforementioned method for detecting the blind area of a vehicle.

Since each module in the detection device 300 provided in this application can be distributed on multiple computers in the same environment or in different environments, this application also provides a system as shown in FIG. 10, which includes multiple computers. Each computer 500 includes a memory 501, a processor 502, a communication interface 503, and a bus 504. Among them, the memory 501, the processor 502, and the communication interface 503 realize the communication connection between each other through the bus 504.

The memory 501 may be a read only memory (Read Only Memory, ROM), a static storage device, a dynamic storage device, or a random access memory (Random Access Memory, RAM). The memory 501 may store computer instructions. When the computer instructions stored in the memory 501 are executed by the processor 502, the processor 502 and the communication interface 503 are used to execute part of the method for detecting the blind area of the vehicle. The memory may also store a data set. For example, a part of the storage resources in the memory 501 is divided into a blind area information library storage module for storing the blind area information library required by the detection device 300.

The processor 502 may adopt a general-purpose central processing unit (Central Processing Unit, CPU), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a graphics processing unit (graphics processing unit, GPU), or any combination thereof. The processor 502 may include one or more chips. The processor 502 may include an AI accelerator, such as a neural network processor (neural processing unit, NPU).

The communication interface 503 uses a transceiver module such as but not limited to a transceiver to implement communication between the computer 500 and other devices or communication networks. For example, the blind spot information can be obtained through the communication interface 503.

The bus 504 may include a path for transferring information between various components of the computer 500 (for example, the memory 501, the processor 502, and the communication interface 503).

Each of the above-mentioned computers 500 establishes a communication path through a communication network. Each computer 500 runs any one or more of the target detection and tracking module 301, the target positioning module 302, the posture determination module 303, and the blind spot determination module 304. Any computer 500 may be a computer in a cloud data center (for example, a server), a computer in an edge data center, or a terminal computing device.

The descriptions of the processes corresponding to each of the above drawings have their respective focuses. For parts that are not described in detail in a certain process, please refer to related descriptions of other processes.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part. The computer program product that realizes the blind spot detection of the vehicle includes one or more computer instructions for detecting the blind spot of the vehicle. When these computer program instructions are loaded and executed on the computer, all or part of the instructions are generated according to the embodiment of the present invention. Figure 5-7 The process or function described.

The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center. Transmission to another website site, computer, server, or data center via wired (such as coaxial cable, optical fiber, digital subscriber line, or wireless (such as infrared, wireless, microwave, etc.)). The computer-readable storage medium stores the implementation A readable storage medium of computer program instructions for detecting the blind spot of a vehicle. The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media integration The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, an SSD).

Claims

A method for detecting the blind area of a vehicle, characterized in that the method includes:

Receiving video data, the video data being captured by a camera set on a traffic road;

Determining the position information and posture information of the vehicle on the traffic road at the current time or in the future according to the video data;

Acquiring blind spot information of the vehicle, where the blind spot information is determined by the structural attributes of the vehicle;

According to the blind spot information of the vehicle, the position information and posture information of the vehicle on the traffic road, the blind spot of the vehicle on the traffic road at the current time or in the future is determined.
The method of claim 1, wherein the method further comprises:

According to the video data and the blind spot of the vehicle on the traffic road at the current time or in the future, it is determined that the vehicle has a blind spot danger, wherein the blind spot danger indicates that the vehicle is in the blind spot memory on the traffic road In other goals;

Send a blind zone alarm.
The method according to claim 2, wherein the blind zone warning includes warning data, and the warning data includes one or more of the following information: the position of the blind zone on the traffic road where the blind zone danger occurs, and Range, location information of the other target on the traffic road, and type of the other target.
The method according to any one of claims 1 to 3, wherein the determining the position information and posture information of the vehicle on the traffic road at a future time according to the video data comprises:

Determining the position information and posture information of the vehicle on the traffic road at the current time according to the video data;

According to the position information and posture information of the vehicle on the traffic road at the current time, predict the position information and posture information of the vehicle on the traffic road at the future time.
The method according to any one of claims 1 to 4, characterized in that the method further comprises: constructing a visual blind spot image according to the blind spot of the vehicle on the traffic road at the current time or the future time;

Send the visual blind spot image.
The method according to any one of claims 1-5, wherein the method further comprises:

Acquiring the driving speed of the vehicle;

Adjusting the blind area of the vehicle on the traffic road at the current time or in the future according to the driving speed of the vehicle.
The method according to claim 2 or 3, wherein the method further comprises:

Sending an adjustment instruction to the vehicle in danger of a blind spot, wherein the adjustment instruction instructs the vehicle to adjust a driving route.
7. The method according to any one of claims 1-7, wherein the method further comprises:

Determine the high-risk blind zone in the blind zone.
8. The method according to any one of claims 1-8, wherein the video data comprises multiple video streams taken by multiple cameras arranged at different positions on the traffic road;

The determining the position information of the vehicle on the traffic road at the current time or the future time according to the video data includes:

Determine the position information of the vehicle in the multiple video streams at a current moment or a future moment according to the multiple video streams;

The position information of the vehicle on the traffic road at the current time or the future time is determined according to the position information of the vehicle in the multiple video streams at the current time or the future time.
The method according to any one of claims 1-9, wherein the blind area information of the vehicle includes: the number of blind areas, the position of each blind area relative to the vehicle, and the shape of the blind area.
A detection device, characterized in that it comprises:

The target detection and tracking module is used to receive video data, the video data being captured by a camera set on a traffic road;

The target positioning module is used to determine the position information of the vehicle on the traffic road at the current time or in the future according to the video data;

An attitude determination module, configured to determine the attitude information of the vehicle on the traffic road at the current time or in the future according to the video data;

The blind spot determination module is used to obtain the blind spot information of the vehicle, and the blind spot information is determined by the structural attributes of the vehicle; it is also used to obtain the blind spot information of the vehicle and the position information of the vehicle on the traffic road And posture information to determine the blind spot of the vehicle on the traffic road at the current time or in the future.
The detection device according to claim 11, wherein:

The blind spot determination module is further configured to determine that the vehicle has a blind spot danger based on the video data and the blind spot of the vehicle on the traffic road at the current or future moments, wherein the blind spot danger indicates the vehicle There are other targets in the blind zone on the traffic road; send a blind zone warning.
The detection device according to claim 12, wherein the blind zone warning includes warning data, and the warning data includes one or more of the following information: the position of the blind zone on the traffic road where the blind zone danger occurs And range, location information of the other target on the traffic road, and the type of the other target.
The detection device according to any one of claims 11-13, wherein:

When the target positioning module is used to determine the position information of the vehicle on the traffic road at a future time according to the video data, it is specifically used to:

Determine the position information of the vehicle on the traffic road at the current time according to the video data;

Predict the location information of the vehicle on the traffic road at a future time according to the location information of the vehicle on the traffic road at the current time;

When the posture determination module is used to determine the posture information of the vehicle on the traffic road at a future time according to the video data, it is specifically used for:

Determine the posture information of the vehicle on the traffic road at the current moment according to the video data;

According to the posture information of the vehicle on the traffic road at the current time, predict the posture information of the vehicle on the traffic road at the future time.
The detection device according to any one of claims 11-14, wherein:

The blind spot determination module is further configured to construct a visual blind spot image according to the blind spot on the traffic road at the current time or in the future; and send the visual blind spot image.
The detection device according to any one of claims 11-15, wherein:

The blind area determination module is also used to obtain the driving speed of the vehicle; adjust the blind area of the vehicle on the traffic road at the current time or in the future according to the driving speed of the vehicle.
The detection device according to claim 12 or 13, wherein:

The blind spot determination module is further configured to send an adjustment instruction to the vehicle in danger of the blind spot, wherein the adjustment instruction instructs the vehicle to adjust a driving route.
The detection device according to any one of claims 11-17, wherein:

The blind zone determination module is also used to determine the high-risk blind zone in the blind zone.
The detection device according to any one of claims 11-18, wherein the video data includes multiple video streams taken by multiple cameras set at different positions on the traffic road;

The target detection and tracking module is further configured to determine the position information of the vehicle in the multiple video streams at a current moment or a future moment according to the multiple video streams;

The target positioning module is further configured to determine the position information of the vehicle on the traffic road at the current time or the future time according to the position information of the vehicle in the multiple video streams at the current time or the future time.
The detection device according to any one of claims 11-19, wherein the blind area information of the vehicle includes: the number of blind areas, the position of each blind area relative to the vehicle, and the shape of the blind area.
An in-vehicle device, the in-vehicle device is arranged on a vehicle, and is characterized in that the in-vehicle device is used to execute the method of any one of claims 1 to 10.
A system, characterized in that the system comprises at least one memory and at least one processor, and the at least one memory is used to store a set of computer instructions;

When the at least one processor executes the set of computer instructions, the system executes the method of any one of claims 1 to 10.
A non-transitory readable storage medium, wherein the non-transitory readable storage medium stores computer program code, and when the computer program code is executed by a computing device, the computing device executes the aforementioned rights The method of any one of 1 to 10 is required.