CN114119955A

CN114119955A - Method and device for detecting potential dangerous target

Info

Publication number: CN114119955A
Application number: CN202010900270.3A
Authority: CN
Inventors: 梁振宝; 张强; 周伟
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-08-31
Filing date: 2020-08-31
Publication date: 2022-03-01

Abstract

The application discloses a method and a device for detecting a potential dangerous target, which are used for accurately and quickly detecting the potential dangerous target and ensuring the driving safety of an automobile, and the method comprises the following steps: acquiring at least two continuous frames of images, wherein the at least two continuous frames of images are obtained by carrying out image acquisition on scenes on two sides of an automobile; determining an ROI area in the first image, and determining motion information of each feature point included in the ROI area, wherein the motion information of each feature point comprises a motion direction of the feature point; the first image is any one of at least two frames of images; clustering each feature point included in the ROI area according to the motion information of each feature point to obtain at least one target located in the ROI area; determining the motion direction of each target according to the motion direction of at least one characteristic point included in each target; and determining the target with the same moving direction as the driving direction of the automobile in the at least one target as a potential dangerous target.

Description

Method and device for detecting potential dangerous target

Technical Field

The present application relates to the field of communications technologies, and in particular, to a method and an apparatus for detecting a potentially dangerous target.

Background

With the increase of the automobile holding capacity, the automobile safety is also valued by more and more people. The Automatic Emergency Brake (AEB) system can ensure the safety of the vehicle to a certain extent. The working principle of the AEB system is as follows: the method comprises the steps that a camera is used for detecting a target existing on a road in front of an automobile, when the distance between the detected target and the automobile does not exceed an alarm distance, the target is considered to have a collision risk, and at the moment, the automobile can automatically adopt a braking measure.

In the prior art, the target detection can be realized based on machine learning, and the detection rate of the complete target positioned in front of the automobile is very high. However, for an object which may be present on both sides of the vehicle and may cut into the front of the vehicle quickly, the detection method of machine learning may be such that the object is not detected or is detected too late. This is because such objects are only partially visible in the captured image and occur in a short time, the detection method of machine learning is difficult to detect when the objects are only partially visible, resulting in detection too late, and the detection result of such objects that are only partially visible also may not detect the objects because it heavily depends on the sample data during machine learning. In the case that the target is not detected or is detected too late, the response time of the AEB function is compressed, resulting in a failure to ensure the driving safety of the vehicle.

Disclosure of Invention

The embodiment of the application provides a method and a device for detecting a potential dangerous target, which are used for quickly and accurately detecting the potential dangerous target so as to respond in advance and ensure the driving safety of an automobile.

In a first aspect, the present application provides a method for detecting a potentially dangerous object, which may be performed by a detection device in an automobile or by a remote server, and sends an execution result to the automobile. The method comprises the following steps: acquiring at least two continuous frames of images, wherein the at least two continuous frames of images are obtained by carrying out image acquisition on scenes on two sides of an automobile; then, an interested ROI area in the first image is determined, and motion information of each feature point included in the ROI area is determined, wherein the motion information of each feature point comprises the motion direction of the feature point; the first image is any one frame image in the at least two frame images; clustering each feature point included in the ROI area according to the motion information of each feature point to obtain at least one target located in the ROI area; determining the motion direction of each target according to the motion direction of at least one characteristic point included in each target; and finally, determining the target with the same motion direction as the driving direction of the automobile in the at least one target as a potential dangerous target.

In a possible scene, an image acquisition device can be installed on the automobile, and the image acquisition device acquires images of scenes on two sides of the automobile and can acquire the continuous at least two frames of images. Of course, the roadside device at the position of the automobile may also be used to acquire images of scenes on two sides of the automobile, and the aforementioned at least two continuous frames of images may also be obtained, which is not limited in this application.

The method has the advantages that the complete targets in front of the automobile can be accurately detected, but the targets which may appear on two sides of the automobile cannot be completely shot by the image acquisition equipment, so that the problems that the targets cannot be detected or the time for detecting the targets is too long can occur when the targets are detected in a machine learning detection mode. In the embodiment of the application, the motion information of each feature point in the ROI area is clustered, the motion direction of the target in the ROI area can be determined, the potential dangerous target can be determined by analyzing the motion direction of the target and the motion direction of the automobile, the detection of the potential dangerous target is obtained by analyzing the motion direction without depending on a detection mode of machine learning, so that the potential dangerous target can be quickly and accurately detected, a response is made in advance, and the driving safety of the automobile is ensured.

In one possible design, when the ROI region in the first image is determined, the ROI region in the first image may be determined according to parameters of an image capturing device for capturing images of scenes on both sides of the vehicle and motion information of the vehicle. The parameters of the image acquisition equipment comprise internal parameters and external parameters, wherein the internal parameters are used for converting between a world coordinate system and an image acquisition equipment coordinate system, and the external parameters are used for converting between the image acquisition equipment coordinate system and a two-dimensional coordinate system; the motion information of the automobile comprises the driving direction and the driving speed of the automobile.

In the design, the ROI area, namely the area where potential dangerous targets possibly exist on two sides of the automobile can be determined through internal and external parameters of the image acquisition equipment and motion information of the automobile, so that the safety of the automobile is ensured.

In one possible design, when the ROI region in the first image is determined according to the parameters of the image capturing device for capturing the images of the scenes on the two sides of the vehicle and the motion information of the vehicle, the longitudinal length of the potential danger region may be determined according to the driving direction and the driving speed of the vehicle and the collision time corresponding to the safe distance of the vehicle; determining the transverse length of the potential danger area according to the width of a lane where the automobile is located; and determining the ROI area in the first image according to the longitudinal length and the transverse length of the potential danger area and the parameters of the image acquisition equipment.

In a possible design, when clustering is performed on each feature point included in the ROI region to obtain at least one target located in the ROI region, specifically, each feature point included in the ROI region may be clustered first to obtain at least one first detection region, where the at least one first detection region is located in the ROI region; then determining first confidence degrees corresponding to the at least one first detection area respectively; first confidence degrees respectively corresponding to the at least one first detection region are determined when clustering all the characteristic points included in the ROI region; then filtering out first detection regions with first confidence degrees smaller than a first confidence degree threshold value in the at least one first detection region to obtain at least one remaining first detection region as at least one second detection region, wherein the first confidence degree threshold value is larger than 0 and smaller than 1; further, according to the determined weight value, weighting the first confidence degrees corresponding to the at least one second detection area respectively to obtain second confidence degrees corresponding to the at least one second detection area respectively; then filtering out second detection regions with second confidence degrees smaller than a second confidence degree threshold value in the at least one second detection region to obtain at least one remaining second detection region, wherein the second threshold value is larger than the first threshold value and smaller than 1; and finally, taking the target respectively included in each of the remaining second detection areas as at least one target located in the ROI area.

In the design, the detection rate of the potential dangerous target can be improved by primarily screening through a lower first confidence threshold, then weighting is carried out according to a weight value, the first confidence is improved to a second confidence, then screening is carried out again through a higher second confidence threshold, the false detection rate can be reduced, and the detection accuracy and the positioning accuracy are improved.

In one possible design, the weight value may be determined according to at least one of the following parameters:

an intersection ratio of the first detection region and the second detection region, wherein the intersection ratio is a ratio of the size of an intersection region of the first detection region and the second detection region to the size of a union region of the first detection region and the second detection region;

the number and position of feature points contained within the second detection region;

the amount of movement of each feature point included in the second detection region;

and the movement direction of each characteristic point contained in the second detection area.

The self-adaptive adjustment of the weight value can be realized through at least one parameter, and the accuracy and the positioning precision of the detection of the potential dangerous target are further ensured.

In a second aspect, embodiments of the present application provide a potentially dangerous object detection apparatus that may function in implementing the first aspect or any of its possible designs. The functions of the above-mentioned potentially dangerous object detection apparatus may be implemented by hardware, or may be implemented by hardware executing corresponding software, where the hardware or software includes one or more modules corresponding to the above-mentioned functions.

Illustratively, the apparatus may include an interface unit and a processing unit. The system comprises an interface unit, a display unit and a display unit, wherein the interface unit is used for acquiring at least two continuous frames of images, and the at least two continuous frames of images are acquired by carrying out image acquisition on scenes on two sides of an automobile; the processing unit is used for determining an interested ROI (region of interest) in a first image and determining motion information of each feature point included in the ROI region, wherein the motion information of each feature point comprises a motion direction of the feature point; the first image is any one frame image in the at least two frame images; clustering all the characteristic points included in the ROI area according to the motion information of all the characteristic points to obtain at least one target positioned in the ROI area; finally, determining the motion direction of each target according to the motion direction of at least one characteristic point included in each target; and determining the target with the same motion direction as the driving direction of the automobile in the at least one target as a potential dangerous target.

In a possible design, the processing unit, particularly when determining the ROI region in the first image, may determine the ROI region in the first image according to parameters of an image capturing device for capturing images of scenes on both sides of the vehicle and motion information of the vehicle. The parameters of the image acquisition equipment can comprise internal parameters and external parameters, wherein the internal parameters are used for converting between a world coordinate system and an image acquisition equipment coordinate system, and the external parameters are used for converting between the image acquisition equipment coordinate system and a two-dimensional coordinate system; the motion information of the automobile comprises the driving direction and the driving speed of the automobile.

In one possible design, when the ROI region in the first image is determined according to parameters of an image capturing device for capturing images of scenes on two sides of the vehicle and motion information of the vehicle, the processing unit may determine a longitudinal length of the potential danger region according to a driving direction and a driving speed of the vehicle and a collision time corresponding to a safe distance of the vehicle; determining the transverse length of the potential danger area according to the width of a lane where the automobile is located; and finally, determining the ROI area in the first image according to the longitudinal length and the transverse length of the potential danger area and the parameters of the image acquisition equipment.

In a possible design, the processing unit is specifically configured to, when clustering feature points included in the ROI area to obtain at least one target located in the ROI area, cluster the feature points included in the ROI area to obtain at least one first detection area, where the at least one first detection area is located in the ROI area; then determining first confidence degrees corresponding to the at least one first detection area respectively; first confidence degrees respectively corresponding to the at least one first detection region are determined when clustering all the characteristic points included in the ROI region; then filtering out first detection regions with a first confidence coefficient smaller than a first confidence coefficient threshold value in the at least one first detection region to obtain at least one remaining first detection region as at least one second detection region, wherein the first confidence coefficient threshold value is larger than 0 and smaller than 1; further, according to the determined weight value, weighting the first confidence degrees corresponding to the at least one second detection area respectively to obtain second confidence degrees corresponding to the at least one second detection area respectively; then filtering out second detection regions with second confidence degrees smaller than a second confidence degree threshold value in the at least one second detection region to obtain at least one remaining second detection region, wherein the second threshold value is larger than the first threshold value and smaller than 1; and finally, taking the target respectively included in each of the remaining second detection areas as at least one target located in the ROI area.

In a third aspect, embodiments of the present application provide a potentially dangerous object detection apparatus, which may have the functionality of any one of the possible designs of the first aspect or the first aspect described above. The functions of the above-mentioned potentially dangerous object detection apparatus may be implemented by hardware, or may be implemented by hardware executing corresponding software, where the hardware or software includes one or more modules corresponding to the above-mentioned functions.

The device comprises at least one processor and at least one memory. The at least one processor is coupled to the at least one memory and is operable to execute computer program instructions stored in the memory to cause the apparatus to perform the method of the first aspect or any of the possible designs of the first aspect. Optionally, the apparatus further comprises a communication interface, the processor being coupled to the communication interface. When the device is a server, the communication interface may be a transceiver or an input/output interface; when the device is a chip included in a server, the communication interface may be an input/output interface of the chip. Alternatively, the transceiver may be a transmit-receive circuit and the input/output interface may be an input/output circuit.

In a fourth aspect, an embodiment of the present application provides a chip system, including: a processor coupled to a memory for storing a program or instructions which, when executed by the processor, cause the system-on-chip to implement the method of the first aspect or any of the possible designs of the first aspect.

Optionally, the chip system further comprises an interface circuit for receiving the code instructions and transmitting them to the processor.

Optionally, the number of processors in the chip system may be one or more, and the processors may be implemented by hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

Optionally, the memory in the system-on-chip may also be one or more. The memory may be integrated with the processor or may be separate from the processor, which is not limited in this application. For example, the memory may be a non-transitory processor, such as a read only memory ROM, which may be integrated with the processor on the same chip or separately disposed on different chips, and the type of the memory and the arrangement of the memory and the processor are not particularly limited in this application.

In a fifth aspect, an embodiment of the present application provides a potentially dangerous target detection apparatus, including a processor and an interface circuit; the interface circuit is used for receiving code instructions and transmitting the code instructions to the processor; the processor is configured to execute the code instructions to perform the method of the first aspect or any of the possible designs of the first aspect.

In a sixth aspect, embodiments of the present application provide a computer-readable storage medium having stored thereon a computer program or instructions which, when executed, cause a computer to perform the method of the first aspect or any one of the possible designs of the first aspect.

In a seventh aspect, an embodiment of the present application provides a computer program product, which when read and executed by a computer, causes the computer to perform the method of the first aspect or any one of the possible designs of the first aspect.

For technical effects that can be achieved by any one of the second aspect to the seventh aspect and any one of the possible designs of any one of the second aspect to the seventh aspect, please refer to the description of the technical effects that can be brought by the corresponding designs of the first aspect and the first aspect, and details are not repeated herein.

Drawings

FIG. 1 is a schematic diagram of a potentially dangerous target provided by an embodiment of the present application;

FIG. 2a is a schematic diagram of a target detection architecture;

FIG. 2b is a schematic view of a target detection process;

fig. 3, fig. 7, and fig. 8 are schematic flow charts of detection of a potentially dangerous target according to an embodiment of the present disclosure;

FIG. 4 is a schematic illustration of a transformation between coordinate systems;

FIG. 5 is a schematic view of an ROI provided in an embodiment of the present application;

FIG. 6 is a schematic diagram illustrating a direction of motion of an object according to an embodiment of the present disclosure;

fig. 9 and fig. 10 are schematic structural diagrams of a potentially dangerous target detection apparatus according to an embodiment of the present application.

Detailed Description

The application provides a method and a device for detecting a potential dangerous target, aiming at more quickly and accurately realizing the detection of the potential dangerous target, so as to respond in time and ensure the safety of an automobile. The method and the device are based on the same technical conception, and because the principles of solving the problems of the method and the device are similar, the implementation of the device and the method can be mutually referred, and repeated parts are not repeated.

Some terms of the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

1) The potential dangerous targets are pointed out to be arranged on two sides of the automobile and can be quickly switched to targets in front of the automobile, and the targets can cause potential safety hazards and affect the safety of the automobile. As shown in fig. 1, the potential dangerous targets are marked as dashed lines, the driving directions of the automobile and the potential dangerous targets are shown as arrow directions, and the potential dangerous targets are located on two sides of the automobile, so that an automobile driver cannot timely sense the existence of the potential dangerous targets and cannot timely respond to the potential dangerous targets which are cut into the front of the automobile quickly at a short distance (such as braking or avoiding) to cause potential safety hazards and influence the automobile safety, and therefore, how to quickly and accurately detect the potential dangerous targets and enable the automobile driver to timely or early respond is very important.

The object or the potentially dangerous object may be a vehicle, which may be a motor vehicle (i.e., an automobile) and/or a non-motor vehicle (e.g., an electric bicycle as shown in fig. 1), and/or a pedestrian.

The direction of travel of the target is the same as or different from the direction of travel of the vehicle, e.g., the direction of travel of the target is the same as the direction of travel of the vehicle in fig. 1. When the driving direction of the target is the same as the driving direction of the automobile and the potential dangerous targets are positioned on two sides of the automobile, the target is not easy to observe by automobile drivers, and when the target is cut into the front of the automobile quickly in a short distance, the automobile drivers are not easy to respond in time, so that the potential safety hazard is high.

The automobile referred to in the embodiments of the present application mainly refers to an automobile in a driving state.

For the sake of convenience of distinction, in the embodiments of the present application, the traveling direction of the automobile is also referred to as a traveling direction, the traveling speed of the automobile is also referred to as a traveling speed, the traveling direction of the target is also referred to as a moving direction, and the traveling speed of the target is also referred to as a moving speed.

2) Coordinate system involved in the image processing: world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system.

The world coordinate system is an absolute coordinate system used to describe the real position of the camera and the object in three-dimensional space, and the unit is meter (m). The world coordinate system may also be regarded as a reference coordinate system. The world coordinate system may be represented as O_w-X_wY_wZ_wIn which O is_wIs the origin of the world coordinate system, X_w、Y_wAnd Z_wThree coordinate axes constituting the world coordinate system. The coordinates of a point in the world coordinate system may be expressed as (X, Y, Z) or (X)_w,Y_w,Z_w)。

The camera coordinate system is a three-dimensional rectangular coordinate system which is established by taking a focusing center of the camera (namely an optical center of the camera) as an origin and taking an optical axis of the camera as a Z axis, and the unit is m. The camera coordinate system may be denoted as O_c-X_cY_cZ_cIn which O is_cIs the origin of the camera coordinate system, X_c、Y_cAnd Z_cThree coordinate axes constituting the camera coordinate system. The coordinates of a point in the camera coordinate system may be expressed as (X)_c,Y_c,Z_c)。

The image coordinate system is a two-dimensional coordinate system established by taking the optical center of the camera as an origin, the optical center of the camera is the image midpoint, and the unit is millimeter (mm). The image coordinate system may be denoted o-xy, where o is the origin of the image coordinate system and x and y constitute two coordinate axes of the image coordinate system. The coordinates of a certain point in the image coordinate system may be denoted as (x, y). f denotes the focal length of the camera, and is O and O_cThe distance of (c).

The pixel coordinate system is a two-dimensional coordinate system established by taking the upper left corner of the image as an origin, and the unit is a pixel (pixel). The two coordinate axes of the pixel coordinate system are composed of u and v. The coordinates of a certain point in the pixel coordinate system may be identified as (u, v).

These coordinate systems can be converted to each other.

It should be understood that, in the embodiment of the present application, only the "camera" is taken as an example, and the type of the image capturing device is not limited, and the "camera" may also be replaced by other types of image capturing devices, such as a camera, a car recorder, and the like.

3) The camera parameters comprise external parameters and internal parameters, the external parameters are used for determining the relative position relationship between the world coordinate system and the camera coordinate system, and the internal parameters are used for determining the projection relationship of the camera from the three-dimensional space to the two-dimensional image. That is, the parameters of the camera may enable the conversion between the above coordinate systems.

4) The region of interest (ROI) is an image region selected in the image, and the content of the ROI is more interesting or interesting than the content of other regions in the image, and can be a focus of attention in the image processing process. The content in the region of interest is the object of interest, including in the present embodiment a potentially dangerous object.

"and/or" in the present application, describing an association relationship of associated objects, means that there may be three relationships, for example, a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

The plural in the present application means two or more.

In addition, it is to be understood that the terms first, second, etc. in the description of the present application are used for distinguishing between the descriptions and not necessarily for describing a sequential or chronological order.

In order to facilitate understanding of the embodiments of the present application, an application scenario used in the present application is described first.

With the increase of the automobile holding capacity, the automobile safety is also valued by more and more people. At present, in order to ensure the safety of automobiles, automobiles are all provided with safety belts and safety air bags, and active safety systems such as AEB (automatic enforcement) and Lane Keeping Assistance (LKA) are gradually becoming the standard of automobiles. The working principle of the AEB system is as follows: the method comprises the steps that a camera is used for detecting a target existing on a road in front of an automobile, when the distance between the detected target and the automobile does not exceed an alarm distance, the target is considered to have a collision risk, and at the moment, the automobile can automatically adopt a braking measure.

In the prior art, the target detection can be realized based on machine learning, and the detection rate of the complete target positioned in front of the automobile is very high. However, for an object which may be present on both sides of the vehicle and may cut into the front of the vehicle quickly, the detection method of machine learning may be such that the object is not detected or is detected too late. This is because such objects are only partially visible in the captured image and occur in a short time, the detection method of machine learning is difficult to detect when the objects are only partially visible, resulting in detection too late, and the detection result of such objects that are only partially visible also may not detect the objects because it heavily depends on the sample data during machine learning. In the case where the target is not detected or is detected too late, the response time of the AEB function is compressed, resulting in a failure to ensure vehicle safety.

In order to be able to respond to potentially dangerous targets in advance, the following approaches have also been proposed in the prior art.

The method comprises the steps of fusing the distance measuring capability of the laser radar and the acquisition function of the visual camera to determine the accurate position of a target, so that the response is realized in advance, and the safety of an automobile is guaranteed.

As shown in fig. 2a, the vehicle includes a vehicle speed sensor, a laser radar, a vision camera, a digital image processor, a data processor, a warning indicator, and a Controller Area Network (CAN) bus. The laser radar is arranged on the roof of the automobile, and can detect targets in front of and on two sides of the automobile and measure the distance between the targets and the automobile and the movement speed of the targets. The visual camera acquires the image of the target, and the digital image processor can process the image of the target to obtain the related information of the target. The data processor can fuse the distance information measured by the laser radar and the related information of the target acquired by the visual camera to determine the accurate position of the target. The vehicle speed sensor CAN acquire the driving speed of the vehicle and send the driving speed to the data processor through the CAN bus, and the data processor determines whether the distance between the target and the vehicle does not exceed the alarm distance according to the accurate position of the target, the movement speed of the target and the driving speed of the vehicle. And if the distance between the target and the automobile does not exceed the alarm distance, early warning is carried out through the early warning indicator so as to realize early response.

In the method, a laser radar is additionally added in the automobile, and the application cost of the existing laser radar is high and is easy to damage, so that the method has limited commercial value, and the automobile-scale laser radar is few and cannot well meet the requirement on automobile safety.

And secondly, combining the detection function of the drivable area of the automobile with the detection function of the target to determine whether the potential dangerous target exists, so that the advanced response is realized, and the safety of the automobile is ensured. As shown in fig. 2b, the method comprises the following steps:

s201: and acquiring a travelable area point set in the image acquired by the front-view camera according to the travelable area detection function. The front view camera may be mounted in the automobile.

S202: and preprocessing the travelable region point set.

S203: clustering the preprocessed pixel points in the travelable region point set, and determining at least one target in the travelable region.

S204: generating coordinates of the at least one target in the travelable region.

S205: and tracking the at least one target, and determining whether a potentially dangerous target which is cut in at a close distance exists. If so, go to S206; if not, returning to the step S201 to continue to acquire the image.

S206: and calculating the acceleration and deceleration of the automobile to be controlled according to different situations.

S207: and carrying out remote control on the automobile in advance according to the movement speed of the potential dangerous target and the driving speed of the automobile so as to realize advanced response.

The detection process of the travelable region in the method is realized based on a deep learning method, a large number of samples are needed, but the marked samples needed by the travelable region detection are difficult to obtain, the marking precision is not high, and therefore the travelable region cannot be detected quickly and accurately, and the method cannot well adapt to the requirement of automobile safety. Moreover, marking samples required by detection of the drivable area are marked manually, and data dependence and application cost are increased.

In summary, several methods proposed in the prior art cannot detect a potentially dangerous target quickly and accurately, so that the driving safety of the vehicle cannot be ensured.

Based on this, the embodiment of the application provides a method and a device for detecting a potentially dangerous target, which are used for quickly and accurately detecting the potentially dangerous target so as to respond in advance and ensure the driving safety of an automobile. As shown in fig. 3, the following steps are included, and the following processes may be performed by the vehicle-mounted device, or may be performed by the road-side device:

s301: the detection module determines the ROI according to the parameters of the camera and the motion information of the automobile. The detection module may be located in the automobile, or may be located in other devices outside the automobile, such as road side devices located on both sides of a driving road of the automobile, and sends the detected information to a remote server for processing, etc.

The camera may be arranged on a car and the camera may take (all or part of) an image of a scene on both sides of the car, e.g. the camera may be a tachograph mounted on the car. The parameters of the camera generally refer to internal and external parameters of the camera, namely, internal parameters and external parameters of the camera. The parameters of the camera are used to realize the conversion between the two-dimensional coordinate system (two-dimensional image/two-dimensional plane) and the world coordinate system (three-dimensional space) where the camera is located. Of course, the camera may not be located on the automobile, and if the camera is disposed in the road side device where the automobile is located, the road side device captures images of scenes on two sides of the automobile.

The motion information of the vehicle includes, but is not limited to, a driving direction and a driving speed of the vehicle. Optionally, the camera may collect continuous multi-frame images, and the detection module determines the driving direction and the driving speed of the vehicle according to the continuous multi-frame images. Or optionally, the power system in the automobile directly acquires the driving direction and the driving speed of the automobile and reports the driving direction and the driving speed to the detection module.

The ROI area is related to motion information of the automobile, wherein the driving direction and the driving speed can determine the size of the ROI area on two sides of the automobile. The two sides of the car refer to the left and right sides of the car, which may be left and right relative to the orientation of the face of a driver of the car when driving the car.

In S301, the detection module may determine sizes of potentially dangerous regions on two sides of the vehicle according to the driving direction and the driving speed of the vehicle, and then determine sizes of ROI regions on two sides of the vehicle according to the sizes of the potentially dangerous regions and the parameters of the camera. Wherein the potentially dangerous region is a region in the three-dimensional space, i.e. in the world coordinate system, and the ROI region is a region in the two-dimensional image, i.e. in the image coordinate system or in the pixel coordinates. For example, the world coordinate system needs to be converted into the camera coordinate system, and then the world coordinate system is projected to an image coordinate system by the camera coordinate system, and the image coordinate system can also be converted into the pixel coordinate system. Fig. 4 is a schematic diagram illustrating a pinhole imaging principle of the camera, where the schematic diagram illustrates a projection relationship between the camera coordinate system and the image coordinate system, a coordinate of a point P in the camera coordinate system is (X, Y, Z), a projection point of the point P in the image coordinate system is a point P, the coordinate of the point P is (X, Y, f), and f represents a focal length of the camera. The conversion formula from the world coordinate system to the pixel coordinate system is shown as formula I, and the internal parameters of the camera are

The external parameter of the camera is

R and T in the camera external parameters respectively represent a rotation matrix and a translation vector converted from the world coordinate system to the camera coordinate system, and u in the camera internal parameters₀And v₀Respectively representing the coordinate values of the origin of the camera coordinate system in the pixel coordinate system and the camera internal reference f_xAnd f_yRespectively representing the focal length of the camera in the x-axis direction (horizontal direction) and the y-axis direction (vertical direction) of the camera coordinate systemUp) component. dx and dy denote the length of one pixel in the x-axis direction and the y-axis direction, respectively, in mm.

For example, if the vehicle is traveling forward, the vehicle is traveling at a speed of 20 kilometers per hour (km/h), and the safe distance of the vehicle corresponds to a Time To Collision (TTC) of 1 second(s), where 1m/s is equal to 3.6km/h, the detection module may determine that the longitudinal length of the potential hazard zone is a safe distance, i.e., the longitudinal length of the potential hazard zone is 20 km/h/3.6 km/h 5.5m, and the lateral length of the potential hazard zone may be a lateral distance (e.g., 3.75m) from the front bumper of the vehicle. The detection module may convert ROI regions of the obtained two-dimensional image according to the potentially dangerous region, for example, ROI regions located on both sides of the vehicle are shown by a dashed box in fig. 5, that is, a region framed by the dashed box is the ROI region.

The ROI region is a region where a potentially dangerous target may appear, which needs attention, that is, the ROI region may include the potentially dangerous target therein. How the detection module detects the potentially dangerous object in the ROI region will be described in detail in the following steps.

The detection module may determine the ROI area in each acquired image. Thus, the ROI area can be dynamically adjusted according to the change of the motion information of the automobile.

In S301, the detection module calculates and generates the ROI according to the internal and external parameters of the camera and the motion information of the vehicle, which can greatly reduce the amount of calculation and reduce noise interference caused by other irrelevant regions, and the detection process of the ROI does not depend on a deep learning method and does not need to label a large number of ROI samples, so that the ROI can be quickly and accurately detected.

S302: the detection module performs feature point extraction in the ROI area.

The process of extracting the feature points can also be regarded as a process of detecting corner points, and the corner points can be understood as pixel points with specific features.

For example, the detection of the corner may be performed by a method such as HARRIS corner detection or FAST corner detection. The basic principle of the HARRISS corner point detection method is as follows: and (3) sliding a fixed window in any direction in the image, comparing the gray change degrees of pixel points in corresponding windows before and after sliding, and if the sliding in any direction has large gray change, determining that an angular point exists in the window. The basic principle of the FAST corner detection method is as follows: if a certain pixel point is in a different region from enough pixel points in the surrounding region, the pixel point may be an angular point, and in the gray-scale image, the gray-scale value of the pixel point is greater than or less than the gray-scale value of the enough pixel points in the surrounding region.

The feature points extracted in S302 may belong to the potentially dangerous target, and may not belong to the potentially dangerous target.

The detection module can extract the characteristic points in the ROI area of each frame of image.

Optionally, after extracting the one or more feature points, the detection module may further determine the positions of the feature points in the two-dimensional image. The position of the feature point in the two-dimensional image may be represented by coordinates in the image coordinate system or the pixel coordinate system, and in the embodiment of the present application, the coordinates in the image coordinate system are mainly used as an example for description.

S303: the detection module determines inter-frame motion information for feature points within the ROI area.

In S303, the detection module may determine inter-frame motion information of the feature points in the ROI region in the image according to the continuous (at least) two frames of images. For example, the detection module determines that the ROI region in the current frame includes a first feature point, and the ROI region in the previous frame image of the current frame also includes the first feature point, it is understood that the first feature point in the previous frame image is a feature point corresponding to the first feature point in the current frame image, and the feature point corresponding to the first feature point refers to a feature point corresponding to the first feature point position and/or a feature point corresponding to a feature possessed by the first feature point. The position of the first feature point in the previous frame image is (x1, y1), the position of the first feature point in the current frame image is (x2, y2), the detection module determines the inter-frame motion of the first feature point to be (x1, y1) to (x2, y2), and the detection module may determine the direction and speed of the inter-frame motion of the first feature point according to (x1, y1) and (x2, y 2). For example, the detection module may determine the direction of the inter-frame movement of the first feature point according to the directions of (x1, y1) to (x2, y2), the detection module may determine the speed of the inter-frame movement of the first feature point according to the distance between (x1, y1) and (x2, y2) and the acquisition time interval between two frames of images, and the detection module may determine the size of the inter-frame movement of the first feature point, that is, the motion amount, according to the distance between (x1, y1) and (x2, y 2).

The detection module may track a position of a first feature point in the previous frame image based on an illumination invariance assumption between two adjacent frames according to a pyramid optical flow algorithm (Lukas-Kanade optical flow algorithm), so as to obtain a position of the first feature point in the current frame image.

S304: the detection module acquires motion information of the automobile.

The motion information of the automobile comprises the driving direction and the driving speed of the automobile, and can also comprise information such as the gear position and the steering wheel angle of the automobile.

In S304, the detection module may obtain the motion information of the vehicle through a CAN bus.

S305: the clustering module clusters the feature points into Bounding Boxes (BBOX), and feeds the BBOX back to the detection module. The detection module may filter the BBOX having a first confidence level less than a first confidence level threshold based on the first confidence level of the BBOX.

In S305, the clustering module may cluster the feature points in the ROI to obtain at least one BBOX, where each BBOX includes a target. That is, at least one object is clustered in the ROI area. Generally, due to the complexity of the actual scene, multiple BBOX, multiple targets can be clustered in the ROI region. Each BBOX is a detection frame, and the region framed by the detection frame is a detection region, so in the embodiment of the present application, the detection frame and the detection region can be equally understood, and the concepts of the detection frame and the detection region can be alternatively used.

In S305, the clustering module may perform clustering with a minimum bounding rectangle according to the feature points meeting the condition, where the BBOX is the minimum bounding rectangle. Since the feature points are not uniformly distributed, the clustered BBOX is not necessarily accurate, but has a guiding meaning for determining a potentially dangerous target in a subsequent step. Since the clustered BBOX is not necessarily accurate, BBOX can also be considered as a clustered rough BBOX.

Wherein the plurality of feature points clustered into one BBOX may have one or more of the following features: the movement speed of the characteristic points is greater than or equal to the driving speed of the automobile, the movement direction of the characteristic points is the same as the driving direction of the automobile, the movement direction of the characteristic points is opposite to the driving direction of the automobile, or the number of the characteristic points exceeds a first number. The first number is a minimum value of the number of feature points included in a predetermined BBOX, and may define the size of the BBOX region. For example, the feature points meeting the condition are feature points of a motion state of which the motion state meets a potentially dangerous target, that is, the motion direction of the feature points meeting the condition is the same as the driving direction of the automobile, and the motion speed is greater than or equal to the driving speed of the automobile. In general, a plurality of feature points clustered to one BBOX belong to the same target, and potentially dangerous targets may be included in the BBOX. The BBOX of the potentially dangerous target may be shown in fig. 1 as a dashed box.

For targets which are positioned on two sides of the automobile and cut into the front of the automobile quickly in a close range, automobile drivers are less prone to being observed and respond in time, therefore, the targets can be used as potential dangerous targets, and the moving direction of the potential dangerous targets is the same as the driving direction of the automobile. Objects that differ from the direction of travel of the vehicle are easily observed by the driver of the vehicle, and therefore such objects can be considered as non-potentially dangerous objects, i.e. objects that do not potentially danger are moving in a direction that differs from the direction of travel of the vehicle. Optionally, as shown in fig. 6, the potentially dangerous target and the non-potentially dangerous target may be marked in different marking manners in the image, so as to be more clearly distinguished, for example, the potentially dangerous target and the non-potentially dangerous target may be marked by marking frames with different colors, and/or an arrow corresponding to the moving direction of the potentially dangerous target is different in color from an arrow corresponding to the moving direction of the non-potentially dangerous target.

The BBOX includes location information of the BBOX, category information to which an object in the BBOX belongs, and a first confidence degree that belongs to the category, where the confidence degree of the object is also used to represent the confidence degree of the BBOX, and the first confidence degree may be regarded as a confidence degree that the object in the BBOX belongs to the category, and the confidence degree is generally expressed by a probability, and a value interval of the first confidence degree is [0,1 ].

In the embodiment of the present application, at least two confidence thresholds are set, taking setting two confidence thresholds as an example, including a first confidence threshold and a second confidence threshold, where the first confidence threshold is smaller than the second confidence threshold, the first confidence threshold is not smaller than 0, and the second confidence threshold is not larger than 1. By setting a lower first confidence threshold (hereinafter referred to as a low threshold) in this S305, more BBOX can be released, improving the detection rate of potentially dangerous targets.

In S305, the detecting module may detect BBOX in the ROI region by sliding a window according to the machine learning and deep learning techniques, and the clustering module may be trained by the machine learning and deep learning techniques. Or the detection module may input the image (marked with the ROI region in the image) into a neural network to obtain BBOX, in which case the clustering module may be a neural network architecture-based clustering model.

For the convenience of distinguishing, BBOX obtained by clustering by the clustering module is represented by Bp, and BBOX obtained by screening through a first confidence coefficient threshold value is represented by Bi.

S306: and the detection module increases the confidence coefficient of the BBOX to a second confidence coefficient, filters the BBOX with the second confidence coefficient smaller than a second confidence coefficient threshold value, and outputs a final detection result.

The detection module can improve the confidence of the BBOX in a weighting mode, and further screen by adopting a higher second confidence threshold (hereinafter referred to as a high threshold), so that the false detection rate is reduced, and the detection accuracy rate is improved.

For example, the detection module increases the BBOX from a first confidence { Bi } to a second confidence { Wi x Bi }, the second confidence having an interval of values of [0,1 ]. Wherein Wi is an adaptive weight, and can be adjusted according to at least one of the following parameters:

the first parameter is as follows: an intersection over Intersection (IOU) ratio of Bi and Bp, where Bi may be BBOX obtained by screening through a first confidence threshold, and Bp may be BBOX obtained by clustering feature points through a clustering module. The intersection and combination ratio of Bi and Bp is the ratio of the size of the intersection region of Bi and Bp to the size of the combination region of Bi and Bp. The higher the IOU value is, the higher the classification confidence of Bi is, the higher the positioning accuracy of Bi is, and the larger the corresponding Wi value is.

And a second parameter: the number and positions of the feature points contained in the Bi, and the positions of the feature points can be used for representing the Bi positioning accuracy. The more the number of the characteristic points is, the higher the classification confidence coefficient of Bi is, the higher the positioning accuracy of Bi is, and the larger the corresponding Wi value is.

And (3) parameters III: the motion directions of the feature points contained in Bi are different from each other, and the contribution of the different motion directions of the feature points to the weighting coefficient Wi is different. The more the characteristic point that corresponds to the movement direction of the cut-in target (i.e., the potentially dangerous target) contributes to Wi, the greater the contribution of the characteristic point that has the same movement direction as the driving direction of the automobile, for example, to Wi, and the greater the corresponding Wi value.

And a fourth parameter: the motion amounts of the feature points included in Bi are different from each other in the contribution to Wi. The larger the movement amount of the characteristic point is, the larger the contribution to Wi is, the smaller the movement amount of the characteristic point is, the smaller the contribution to Wi is, and therefore the limited detection of the target of fast cut-in can be guaranteed.

In S306, by increasing the confidence of the candidate detection region and setting a high threshold, the false detection rate can be reduced, and the detection accuracy and the positioning accuracy can be improved.

In S306, the detection module may further screen the weighted classification confidence and positioning accuracy, and then perform non-maximum suppression (NMS) to obtain a final detection result. For example, the NMS may further screen the obtained BBOX according to the weighted classification confidence and localization accuracy, and output a final detection result. The score threshold may be any value, and is not limited in the embodiments of the present application.

Where S305 and S306 can be considered as detecting within the ROI using dual thresholds, including a low threshold Tlow and a high threshold Thigh, where 0< Tlow < Thigh < 1. It will be appreciated that the method of dual threshold detection may also be applied directly in the acquired image. The process of dual threshold detection can be seen in fig. 7, comprising the following steps:

s701: the detection module inputs the acquired image into a neural network, and obtains an output first detection area based on the neural network.

The neural network may implement the function of the clustering module in S305, and the first detection area is BBOX obtained by clustering in S305.

S702: the detection module determines whether the first detection region is located within the ROI region. S703 is performed for a first detection region located within the ROI, and S705 is performed for a first detection region not located within the ROI.

S703: and the detection module filters a first detection area with a first confidence coefficient not exceeding Tlow by adopting a low threshold Tlow to obtain a second detection area. The first confidence of the second detection region is equal to or greater than the Tlow.

S704: and the detection module calculates the weight Wi of the second detection area, and weights the first confidence coefficient by adopting the weight Wi to obtain a second confidence coefficient { Wi-Bi } of the second detection area.

S705: and the detection module further filters a second detection area with a second confidence coefficient not exceeding the Thigh by adopting a high threshold Thigh to obtain a third detection area. The second confidence of the third detection region is equal to or greater than Thigh.

S706: and the detection module further filters the third detection area through an NMS algorithm and outputs a final detection result.

Illustratively, the inputs to the NMS algorithm are the locations of the plurality of third detection regions and the second confidence level for each third detection region. The screening conditions of the NMS algorithm comprise: for the overlapped third detection areas, deleting the third detection areas with the second confidence degrees exceeding a third confidence degree threshold value from the overlapped third detection areas, and deleting the third detection areas without exceeding the third confidence degree threshold value; for a third detection area without overlap, the third detection area is retained. The NMS outputs one or more third detection regions that satisfy the deletion screening conditions as final detection regions that include potentially dangerous targets.

In the prior art, the detection rate of a complete target is high, but for a target which is cut into rapidly, the target needs to be detected in advance in the process of entering the field of view (FOV) of a camera, and the target is expected to be detected faster, so that the requirement on the detection capability is higher when the target is shielded more seriously. When detecting a potential dangerous target, the mainstream detection method of machine learning and deep learning depends heavily on sample data, and more calculation power needs to be consumed, so that the precision detection of the detection model on the seriously sheltered target can be ensured, the more calculation power improves higher requirements on system resources, and the detection real-time performance of the system is also reduced. The potentially dangerous target method provided by the embodiment of the application reduces the dependency of the shielding target on the sample, can improve the rapid and accurate detection of the cut-in target under the condition of using less calculation force, and only uses image acquisition equipment without introducing other sensors, thereby reducing the system cost.

With reference to the foregoing embodiments and accompanying drawings, as shown in fig. 8, an embodiment of the present application provides a method for detecting a potentially dangerous target. The method comprises the following steps:

s801: the method comprises the steps that first equipment obtains at least two continuous frames of images, and the at least two continuous frames of images are obtained by carrying out image acquisition on scenes on two sides of an automobile.

The first device may be an automobile or located in an automobile, or the first device may be another device outside the automobile, such as a remote server, and the server performs detection of the potentially dangerous object and then sends the detection result to the automobile.

In a possible scenario, the automobile is provided with an image capturing device, and the image capturing device can capture images of scenes on two sides of the automobile, such as at least two continuous frames of images. In another possible scene, the road side unit at the position of the automobile can also acquire images of scenes on two sides of the automobile, namely, the road side unit is provided with image acquisition equipment.

The image captured by the image capturing device may include a complete target image, and may include a target image that is only partially visible.

S802: the first device determines an ROI (region of interest) in a first image and determines motion information of various feature points included in the ROI, wherein the motion information of each feature point comprises a motion direction of the feature point; the first image is any one of the at least two frames of images.

In S802, the first device may further determine an ROI region in the first image according to the parameters for the image capturing device and the motion information of the car.

The parameters of the image acquisition equipment comprise internal parameters and external parameters, the internal parameters are used for converting between a world coordinate system and an image acquisition equipment coordinate system, and the external parameters are used for converting between the image acquisition equipment coordinate system and a two-dimensional coordinate system; the motion information of the automobile comprises the driving direction and the driving speed of the automobile.

For example, the first device may determine the longitudinal length of the potentially dangerous area according to the driving direction and the driving speed of the automobile and the collision time corresponding to the safe distance of the automobile; determining the transverse length of the potential danger area according to the width of a lane where the automobile is located; and determining the ROI area in the first image according to the longitudinal length and the transverse length of the potential danger area and the parameters of the image acquisition equipment.

S803: and the first equipment clusters all the characteristic points included in the ROI area according to the motion information of all the characteristic points to obtain at least one target positioned in the ROI area.

The first device may cluster a plurality of feature points whose feature information meets a condition.

S804: and the first equipment determines the motion direction of each target according to the motion direction of at least one characteristic point included by each target.

The motion directions of the feature points in the first detection area obtained by clustering are the same, so that the motion direction of the target in the first detection area can be determined by adopting the motion direction of at least one feature point.

S805: and the first equipment determines the target with the same motion direction as the driving direction of the automobile in the at least one target as a potential dangerous target.

In S805, clustering each feature point included in the ROI area to obtain at least one first detection area, where the at least one first detection area is located in the ROI area; determining first confidence degrees corresponding to the at least one first detection area respectively; first confidence degrees respectively corresponding to the at least one first detection region are determined when clustering all the characteristic points included in the ROI region; filtering out first detection regions with a first confidence coefficient smaller than a first confidence coefficient threshold value in the at least one first detection region to obtain at least one remaining first detection region as at least one second detection region, wherein the first confidence coefficient threshold value is larger than 0 and smaller than 1; according to the determined weight value, weighting the first confidence degrees corresponding to the at least one second detection region respectively to obtain the second confidence degrees corresponding to the at least one second detection region respectively; filtering out second detection regions with second confidence degrees smaller than a second confidence degree threshold value in the at least one second detection region to obtain at least one remaining second detection region, wherein the second threshold value is larger than the first threshold value and smaller than 1; and taking the target respectively included in each of the remaining second detection areas as at least one target located in the ROI area.

Wherein the weight value is determined according to at least one of the following parameters:

The specific implementation manner shown in fig. 8 in the embodiment of the present application may be referred to the description of the related embodiments above.

The embodiments of the present application may be used in combination with each other or may be used alone.

The scheme provided by the present application is presented mainly from the perspective of the method flow in fig. 8. It is understood that, in order to implement the above functions, the apparatus may include a corresponding hardware structure and/or software module for performing each function. Those of skill in the art would readily appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the case of an integrated unit, fig. 9 shows a possible exemplary block diagram of a potentially dangerous target detection apparatus, which may be in the form of software 900, involved in the embodiments of the present application. The potentially dangerous target device 900 may include: an interface unit 901 and a processing unit 902.

The potentially dangerous target device 900 may be the automobile described above or may also be a semiconductor chip disposed in an automobile or other equipment. Specifically, in an embodiment, the interface unit 901 is configured to acquire at least two consecutive frames of images, where the at least two consecutive frames of images are obtained by image acquisition of scenes on two sides of an automobile;

the processing unit 902 is configured to determine an ROI region of interest in the first image, and determine motion information of feature points included in the ROI region, where the motion information of each feature point includes a motion direction of the feature point; the first image is any one frame image in the at least two frame images; and

clustering all the characteristic points included in the ROI area according to the motion information of all the characteristic points to obtain at least one target positioned in the ROI area;

determining the motion direction of each target according to the motion direction of at least one characteristic point included in each target; and determining the target with the same motion direction as the driving direction of the automobile in the at least one target as a potential dangerous target.

In one possible design, when the processing unit 902 determines the ROI region in the first image, it is specifically configured to: and determining the ROI area in the first image according to the parameters of image acquisition equipment for acquiring images of scenes on two sides of the automobile and the motion information of the automobile.

In a possible design, when the processing unit 902 determines the ROI region in the first image according to the parameters of the image capturing device for capturing the images of the scenes on the two sides of the automobile and the motion information of the automobile, specifically, the processing unit is configured to: determining the longitudinal length of a potential danger area according to the driving direction and the driving speed of the automobile and the collision time corresponding to the safe distance of the automobile; determining the transverse length of the potential danger area according to the width of a lane where the automobile is located; and determining the ROI area in the first image according to the longitudinal length and the transverse length of the potential danger area and the parameters of the image acquisition equipment.

In a possible design, when the processing unit 902 clusters each feature point included in the ROI region to obtain at least one target located in the ROI region, the processing unit is specifically configured to: clustering all feature points included in the ROI area to obtain at least one first detection area, wherein the at least one first detection area is located in the ROI area; determining first confidence degrees corresponding to the at least one first detection area respectively; first confidence degrees respectively corresponding to the at least one first detection region are determined when clustering all the characteristic points included in the ROI region; filtering out first detection regions with a first confidence coefficient smaller than a first confidence coefficient threshold value in the at least one first detection region to obtain at least one remaining first detection region as at least one second detection region, wherein the first confidence coefficient threshold value is larger than 0 and smaller than 1; according to the determined weight value, weighting the first confidence degrees corresponding to the at least one second detection region respectively to obtain the second confidence degrees corresponding to the at least one second detection region respectively; filtering out second detection regions with second confidence degrees smaller than a second confidence degree threshold value in the at least one second detection region to obtain at least one remaining second detection region, wherein the second threshold value is larger than the first threshold value and smaller than 1; and taking the target respectively included in each of the remaining second detection areas as at least one target positioned in the ROI area.

In one possible design, the weight value is determined according to at least one of the following parameters:

an intersection ratio of the first detection region and the second detection region, the intersection ratio being a ratio of a size of an intersection region of the first detection region and the second detection region to a size of a union region of the first detection region and the second detection region;

The division of the unit in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation. The functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

As shown in fig. 10, the present embodiment also provides a schematic structural diagram of another possible dangerous object detecting apparatus, which includes at least one processor 1002 and at least one communication interface 1004. Further, the potentially dangerous object detecting apparatus may also include a memory 1006, wherein the memory 1006 is used for storing computer programs or instructions. The memory 1006 may be an in-processor memory or an off-processor memory. In the case where the unit modules depicted in fig. 10 are implemented by software, software or program codes required for the processor 1002 to perform the respective actions are stored in the memory 1006. The processor 1002 is configured to execute programs or instructions in the memory 1006 to implement the steps shown in fig. 8 in the above-described embodiment. The communication interface 1004 is used to enable communication between the apparatus and other apparatuses.

In the case where the memory 1006 is disposed outside the processor, the memory 1006, the processor 1002, and the communication interface 1004 are connected to each other by a bus 1008, and the bus 1008 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. It should be understood that the bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 10, but this is not intended to represent only one bus or type of bus.

It should be noted that, for the sake of brevity, the operations and/or functions of the modules in the apparatus 1000 are not described again here in order to implement the corresponding flow of the method shown in fig. 8.

An embodiment of the present application further provides a chip system, including: a processor coupled to a memory for storing a program or instructions that, when executed by the processor, cause the system-on-chip to implement the method of any of the above method embodiments.

Optionally, the system on a chip may have one or more processors. The processor may be implemented by hardware or by software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

The system-on-chip may be, for example, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a system on chip (SoC), a Central Processing Unit (CPU), a Network Processor (NP), a digital signal processing circuit (DSP), a Microcontroller (MCU), a Programmable Logic Device (PLD), or other integrated chips.

It will be appreciated that the steps of the above described method embodiments may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware processor, or may be implemented by a combination of hardware and software modules in a processor.

The embodiment of the present application further provides a computer-readable storage medium, where computer-readable instructions are stored in the computer-readable storage medium, and when the computer-readable instructions are read and executed by a computer, the computer is enabled to execute the method in any of the above method embodiments.

The embodiments of the present application further provide a computer program product, which when read and executed by a computer, causes the computer to execute the method in any of the above method embodiments.

It should be understood that the processor mentioned in the embodiments of the present application may be a Central Processing Unit (CPU), and may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It will also be appreciated that the memory referred to in the embodiments of the application may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable EPROM (EEPROM), or a flash memory. Volatile memory can be Random Access Memory (RAM), which acts as external cache memory. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), double data rate SDRAM, enhanced SDRAM, SLDRAM, Synchronous Link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

It should be noted that when the processor is a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, the memory (memory module) is integrated in the processor.

It should be noted that the memory described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

It should be understood that, in the various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of the processes should be determined by their functions and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for detecting potentially dangerous objects, comprising:

acquiring at least two continuous frames of images, wherein the at least two continuous frames of images are obtained by carrying out image acquisition on scenes on two sides of an automobile;

determining an interesting ROI (region of interest) in a first image, and determining motion information of various feature points included in the ROI, wherein the motion information of each feature point comprises a motion direction of the feature point; the first image is any one frame image in the at least two frame images;

determining the motion direction of each target according to the motion direction of at least one characteristic point included in each target;

and determining the target with the same motion direction as the driving direction of the automobile in the at least one target as a potential dangerous target.

2. The method of claim 1, wherein determining the ROI area in the first image comprises:

determining an ROI (region of interest) in a first image according to parameters of image acquisition equipment for acquiring images of scenes on two sides of the automobile and motion information of the automobile;

3. The method of claim 2, wherein determining the ROI in the first image based on parameters of an image capture device used to image capture the scene on both sides of the car and motion information of the car comprises:

determining the longitudinal length of a potential danger area according to the driving direction and the driving speed of the automobile and the collision time corresponding to the safe distance of the automobile;

determining the transverse length of the potential danger area according to the width of a lane where the automobile is located;

and determining the ROI area in the first image according to the longitudinal length and the transverse length of the potential danger area and the parameters of the image acquisition equipment.

4. The method of any one of claims 1-3, wherein said clustering feature points included in the ROI area to obtain at least one object located in the ROI area comprises:

clustering all feature points included in the ROI area to obtain at least one first detection area, wherein the at least one first detection area is located in the ROI area;

determining first confidence degrees corresponding to the at least one first detection area respectively; first confidence degrees respectively corresponding to the at least one first detection region are determined when clustering all the characteristic points included in the ROI region;

filtering out first detection regions with a first confidence coefficient smaller than a first confidence coefficient threshold value in the at least one first detection region to obtain at least one remaining first detection region as at least one second detection region, wherein the first confidence coefficient threshold value is larger than 0 and smaller than 1;

according to the determined weight value, weighting the first confidence degrees corresponding to the at least one second detection region respectively to obtain the second confidence degrees corresponding to the at least one second detection region respectively;

filtering out second detection regions with second confidence degrees smaller than a second confidence degree threshold value in the at least one second detection region to obtain at least one remaining second detection region, wherein the second threshold value is larger than the first threshold value and smaller than 1;

and taking the target respectively included in each of the remaining second detection areas as at least one target positioned in the ROI area.

5. The method of claim 4, wherein the weight value is determined according to at least one of the following parameters:

6. A potentially dangerous object detection apparatus, comprising an interface unit and a processing unit:

the interface unit is used for acquiring at least two continuous frames of images, and the at least two continuous frames of images are obtained by carrying out image acquisition on scenes on two sides of the automobile;

the processing unit is used for determining an interested ROI (region of interest) in a first image and determining motion information of each feature point included in the ROI region, wherein the motion information of each feature point comprises a motion direction of the feature point; the first image is any one frame image in the at least two frame images; and

7. The apparatus as claimed in claim 6, wherein the processing unit, when determining the ROI area in the first image, is specifically configured to:

8. The apparatus according to claim 7, wherein the processing unit, when determining the ROI region in the first image based on the parameters of the image capturing device for image capturing of the scenes on both sides of the vehicle and the motion information of the vehicle, is specifically configured to:

9. The apparatus according to any one of claims 6 to 8, wherein the processing unit is configured to cluster the feature points included in the ROI region, and when obtaining the at least one target located in the ROI region, to:

and taking the target respectively included in each of the remaining second detection areas as at least one target located in the ROI area.

10. The apparatus of claim 9, wherein the weight value is determined according to at least one of the following parameters:

11. A potentially dangerous object detection apparatus comprising at least one processor coupled with at least one memory:

the at least one processor configured to execute computer programs or instructions stored in the at least one memory to cause the apparatus to perform the method of any of claims 1-5.

12. A potentially dangerous object detection apparatus comprising a processor and interface circuitry;

the interface circuit is used for receiving code instructions and transmitting the code instructions to the processor;

the processor to execute the code instructions to perform the method of any one of claims 1 to 5.

13. A computer program storage medium having stored thereon computer program instructions which, when executed, cause the method of any one of claims 1 to 5 to be carried out.