CN116863445A - Collision prediction time calculation method and distributed vision collision sensing system - Google Patents
Collision prediction time calculation method and distributed vision collision sensing system Download PDFInfo
- Publication number
- CN116863445A CN116863445A CN202310849733.1A CN202310849733A CN116863445A CN 116863445 A CN116863445 A CN 116863445A CN 202310849733 A CN202310849733 A CN 202310849733A CN 116863445 A CN116863445 A CN 116863445A
- Authority
- CN
- China
- Prior art keywords
- collision
- prediction time
- collision prediction
- current image
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 30
- 238000001514 detection method Methods 0.000 claims abstract description 59
- 238000000034 method Methods 0.000 claims abstract description 30
- 230000000007 visual effect Effects 0.000 claims abstract description 21
- 238000012545 processing Methods 0.000 claims description 37
- 238000004891 communication Methods 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 11
- 238000009434 installation Methods 0.000 claims description 8
- 238000010586 diagram Methods 0.000 description 15
- 230000008859 change Effects 0.000 description 7
- 238000013507 mapping Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 5
- 230000008447 perception Effects 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000004927 fusion Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 241001465754 Metazoa Species 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention provides a collision prediction time calculation method and a distributed visual collision sensing system. The collision prediction time calculation method comprises the following steps: step S1, acquiring a current image shot by a camera; step S2, identifying an object in the current image based on a pre-trained target detection model; step S3, judging whether an object in an expansion mode exists in the current image: if not, returning to the execution step S1; if so, calculating collision prediction time according to the size expansion speed of the object in the expansion mode in the current image and the historical frame image, and executing step S4; and S4, outputting collision prediction time of all the objects in the expansion mode in the current image, and returning to the step S1. According to the method, the depth distance of the three-dimensional space is not required to be acquired, the collision prediction time can be calculated by only utilizing the size expansion speed of the object on the image, and the method has the advantages of being convenient, efficient and accurate in sensing the front obstacle.
Description
Technical Field
The invention relates to the technical field of collision sensing, in particular to a collision prediction time calculation method and a distributed visual collision sensing system.
Background
With the rapid development of artificial intelligence, artificial intelligence technology for simulating human brain and eyes to realize collision perception is also a new technology. Currently, there are two main collision sensing schemes:
firstly, based on multi-sensor fusion, the depth and position information of the obstacle are detected through sensor fusion such as laser radar, ultrasonic radar, GPS and the like, so that collision sensing is realized. Chinese patent CN110834627B (millimeter wave radar-based vehicle collision warning control method, system and vehicle) proposes a millimeter wave radar-based vehicle collision warning control method, which uses obstacle information identified by the millimeter wave radar; and analyzing the longitudinal and transverse distance and speed information, and judging whether collision risk exists between the vehicle and the obstacle according to the distance and speed information. However, in this scheme, the use of the millimeter wave radar is susceptible to external factors such as weather, temperature, rain and snow, radio frequency electromagnetic wave, etc., which result in inaccurate ranging results, and thus the collision risk cannot be accurately determined. Meanwhile, the radar mode is high in cost, and in principle, defects exist, so that effective classification of obstacles cannot be performed, and a large number of misjudgment can be caused, so that the system is in an extremely unstable state.
Secondly, based on vision, surrounding images are acquired through a high-definition camera, and based on a high-performance chip, the depth and position information of the front obstacle are estimated through algorithms such as panoramic stitching, deep learning and SLAM, so that collision sensing is realized. Chinese patent CN112349144B (a vehicle collision early warning method and system based on monocular vision) proposes that first, image data of a view field in front of a vehicle is obtained by using a monocular camera, target detection is performed on the image data, a collision risk area range is set, and targets in the collision risk area are filtered; then, estimating the distance between the vehicle and the front target by utilizing the width of the real object and the focal length of the camera, and estimating the time required for the vehicle to collide with the target by combining the speed information and the acceleration information of the vehicle; and finally, integrating the nearest target distance estimated value and the estimated value of the time required by the vehicle to collide with the target, and carrying out auxiliary early warning on the possible collision situation of the vehicle in the driving process of the vehicle. However, the most important ranging process of the scheme is that the real width of the obstacle in the three-dimensional world must be known in advance, and the real width of different obstacles in the three-dimensional world cannot be accurately obtained in real time in the practical application process, which results in poor applicability of the scheme and poor ranging accuracy and stability.
In summary, most of the existing two mainstream collision sensing schemes, whether based on sensor fusion or vision, need to measure three-dimensional depth and distance, and require expensive hardware support and higher development cost, and stability and accuracy are to be improved.
Disclosure of Invention
The invention aims to solve the technical problems in the prior art and provides a collision prediction time calculation method and a distributed visual collision sensing system.
In order to achieve the above object of the present invention, according to a first aspect of the present invention, there is provided a collision prediction time calculation method including: step S1, acquiring a current image shot by a camera; step S2, identifying an object in the current image based on a pre-trained target detection model; step S3, judging whether an object in an expansion mode exists in the current image: if not, returning to the execution step S1; if so, calculating collision prediction time according to the size expansion speed of the object in the expansion mode in the current image and the historical frame image, and executing step S4; and S4, outputting collision prediction time of all the objects in the expansion mode in the current image, and returning to the step S1.
The technical scheme is as follows: according to the collision prediction time calculation method, objects in an image are identified by utilizing a target detection model, whether the objects in an expansion mode exist or not is judged in the objects, if the objects in the expansion mode exist, the collision prediction time is calculated according to the size expansion speed of the objects in a current image and a historical frame image, and the collision prediction time of all the objects in the expansion mode in the current image is output.
In a preferred embodiment of the present invention, in step S3, the process of determining whether an object in the expansion mode exists in the current image includes: identifying an object which exists in the current image and the historical frame image at the same time, and marking the object as a first object; calculating the sizes of the first object in the current image and the historical frame image respectively; and if the size of the first object in the current image is larger than the size of the first object in the historical frame image, the first object is considered to be in the expansion mode.
The technical scheme is as follows: and tracking the object in the current image and the historical frame image, wherein the object in the expansion mode has the characteristic that the size of the object in the current image is larger than that of the object in the historical frame image, and can be rapidly identified according to the characteristic according to the size change of the object in the two frame images, so that the processing efficiency is improved.
In a preferred embodiment of the present invention, the history frame image is a last frame image captured by a camera.
The technical scheme is as follows: the historical frame image is the last frame image shot by the camera, the characteristic that the collision risk object has large size change in the front frame and the rear frame is fully utilized, and the expansion mode object is more accurately identified.
In a preferred embodiment of the present invention, in step S3, the process of calculating the collision prediction time of the object in the expansion mode is: calculating the size expansion speed:wherein S is p Representing the size of an object in the expanded mode in the history frame image, S c Representing the size of the object in the inflated mode in the current image; calculating collision prediction time:wherein delta is t Representing the acquisition time difference between the historical frame image and the current image.
The technical scheme is as follows: the size ratio of the first object in the historical image frame and the current image is used as the size expansion speed, the size expansion speed is substituted into a pre-built collision prediction time calculation model (TTC calculation formula) to obtain the collision prediction time, and the collision prediction time calculation model can perfectly embody the physical characteristics that the larger the size expansion speed is, the smaller the collision time is, so that the method has practicability.
In order to achieve the above object of the present invention, according to a second aspect of the present invention, there is provided a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements a visual collision prediction time calculation method according to the first aspect of the present invention.
The technical scheme is as follows: the method for calculating the collision prediction time has the technical effects of the first aspect of the invention.
In order to achieve the above object of the present invention, according to a third aspect of the present invention, there is provided an electronic apparatus comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the method of calculating a visual impact prediction time according to the first aspect of the invention.
The technical scheme is as follows: the method for calculating the collision prediction time has the technical effects of the first aspect of the invention.
To achieve the above object of the present invention, according to a fourth aspect of the present invention, there is provided a distributed visual impact sensing system comprising: the cameras are arranged outside the protected main body in a linear array or an area array manner; the plurality of edge processing units respectively correspond to the cameras, respectively acquire output images of the corresponding cameras and execute the steps of the collision prediction time calculation method of the first aspect of the invention to acquire the collision prediction time of each camera detection area, and the edge processing units send the acquired collision prediction time to the central processing unit; and the central processing unit is used for receiving the collision prediction time sent by all the edge processing units, and outputting a collision alarm signal if at least one collision prediction time is smaller than or equal to a time threshold value.
The technical scheme is as follows: the system has the technical effects of the collision prediction time calculation method according to the first aspect of the invention, and also has the following steps: the wide-angle cameras are arranged in a linear array or area array mode, so that the visual field of a protected main body can be increased to the greatest extent, the visual field blind area of the protected main body is reduced, and the perception capability of collision risk of the protected main body is greatly improved; compared with a radar scheme, the method has the advantages of strong anti-interference performance, low cost and simple development, can distinguish collision object types, improves collision sensing accuracy, and reduces false detection rate of collision sensing; the collision risk can be perceived without distance measurement, the collision perception stability is improved, and the consumption of system resources and the requirements of hardware performance are reduced. The camera is flexible, simple and convenient in installation and arrangement mode, wide in application scene, and suitable for various collision scenes needing to be perceived, and the protected main body is not limited to vehicles, robots and the like.
In a preferred embodiment of the present invention, the central processing unit specifically performs: obtaining detection areas of the cameras according to the installation positions of the cameras outside the protected main body, and distributing detection area numbers for the detection areas of each camera; the collision prediction time of the detection areas of all cameras is formed into an array; associating the collision prediction time of the detection area of the camera with the detection area number; and outputting a detection area number associated with the collision alarm signal and the collision prediction time when any collision prediction time in the array is less than or equal to a time threshold.
The technical scheme is as follows: the collision prediction time of all the camera detection areas is formed into an array, so that subsequent centralized management and decision are facilitated, and the position of the area with collision risk can be obtained based on the detection area number while the collision alarm signal can be obtained.
In a preferred embodiment of the invention, the system further comprises a communication module and an acousto-optic early warning prompt module, wherein the central processing unit outputs a collision warning signal to the acousto-optic early warning prompt module through the communication module, and the acousto-optic early warning prompt module carries out an acousto-optic warning after receiving the collision warning signal.
The technical scheme is as follows: the audible and visual alarm mode is adopted, so that the alarm effect can be improved, and related personnel can find out in time.
In a preferred embodiment of the invention, the system further comprises a device terminal early warning prompt module, and the central processing unit is connected and communicated with the device terminal early warning prompt module through a communication module.
The technical scheme is as follows: and the notification breadth is improved, and the remote collision risk monitoring is realized.
Drawings
FIG. 1 is a flowchart of a method for calculating a predicted time to collision according to an embodiment of the present invention;
FIG. 2 is a schematic perspective view;
FIG. 3 is a schematic diagram of an electronic device according to another embodiment of the present invention;
FIG. 4 is a block diagram of a distributed visual impact awareness system in accordance with another embodiment of the invention;
FIG. 5 is a schematic diagram of a linear array of cameras according to another embodiment of the present invention;
FIG. 6 is a schematic diagram of an area array arrangement of cameras according to another embodiment of the present invention;
FIG. 7 is a schematic diagram of an array of CPU units according to another embodiment of the invention;
fig. 8 is a schematic flow chart of a distributed visual collision sensing system in an application scene.
Detailed Description
Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the invention.
In the description of the present invention, it should be understood that the terms "longitudinal," "transverse," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like indicate orientations or positional relationships based on the orientation or positional relationships shown in the drawings, merely to facilitate describing the present invention and simplify the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and therefore should not be construed as limiting the present invention.
In the description of the present invention, unless otherwise specified and defined, it should be noted that the terms "mounted," "connected," and "coupled" are to be construed broadly, and may be, for example, mechanical or electrical, or may be in communication with each other between two elements, directly or indirectly through intermediaries, as would be understood by those skilled in the art, in view of the specific meaning of the terms described above.
The invention discloses a collision prediction time calculation method, in an embodiment, a flow diagram of the method is shown in fig. 1, and the method comprises the following steps:
step S1, acquiring a current image shot by a camera.
In this embodiment, the camera is preferably but not limited to a wide-angle camera, which is small in weather, temperature, rain and snow, radio frequency electromagnetic waves, and the angle of view can be arbitrarily configured. After the camera is installed, the coverage area of the field angle of the camera is the detection area, the camera continuously shoots images, and the current image is the current image frame shot by the camera.
And step S2, identifying the object in the current image based on the pre-trained target detection model.
In the present embodiment, the object detection model is preferably, but not limited to, an existing YOLO (full scale you only look once) object detection model or an SSD (full scale Single Shot MultiBox Detector) object detection model. The training process of the target detection model comprises the following steps: the method comprises the steps of constructing an image set, wherein images in the image set are road pictures in actual application scenes, and the objects in the images are marked in a priori category according to actual requirements, wherein the marked category preferably and not only comprises people, animals, motor vehicles, non-motor vehicles and static objects, but also is preferably and not only a traffic sign board or a traffic management control box. Giving semantic information to objects in the image through category labeling; in actual demands, only objects in a lane or adjacent lanes can be detected, objects outside the lane are not detected, and the detection range can be set according to the actual demands; constructing a network structure of a target detection model; dividing the image set into a training set, a testing set and a verification set, training the constructed target detection model by using the training set, testing the trained target detection model by using the testing set, and verifying the trained target detection model by using the verification set.
In step S2, the pre-trained object detection model identifies the objects in the current image according to the prior information, and the number of the identified objects may be 0 or more.
Step S3, judging whether an object in an expansion mode exists in the current image: if the object in the expansion mode does not exist, returning to the step S1; if there is an object in the expansion mode, a collision prediction time is calculated according to the expansion speed of the size of the object in the expansion mode in the current image and the history frame image, and step S4 is performed.
And S4, outputting collision prediction time of all the objects in the expansion mode in the current image, and returning to the step S1. The number of the objects in the expansion mode in the current image may be 0 or more, and it is necessary to output the collision prediction time of all the objects, and when the number of the objects in the expansion mode in the current image is 0, the collision prediction time of the output objects is a preset larger time value.
Fig. 2 discloses a perspective principle schematic diagram, when the camera is fixed, the imaging size of an object in the detection area of the camera in the image plane becomes larger as the distance between the object and the camera is reduced, namely the size of an image of the object in the two-dimensional image plane of the camera is gradually increased as the distance between the object and the camera is reduced. Therefore, based on the perspective principle, whether the object and the camera possibly have collision risks can be judged according to the size change of the image of the same object in the image plane in the image frames shot successively. If the object is closer to the camera, the image of the object in the image plane gradually increases in size, and the object is considered to be in the expansion mode when the image of the object is in this state. Therefore, by combining the historical images shot by the camera in advance, whether the object is in the expansion mode can be judged, if the object in the expansion mode exists in the current image, the collision prediction time of the object is calculated, and if all the objects in the current image are not in the expansion mode, the step S1 can be executed in a returning mode, and the calculation of the collision prediction time of the next frame of image is performed.
In this embodiment, in order to facilitate quick locking of the change in the size of the image of the object in the image captured by the camera, preferably, step S3 further includes an object tracking step of tracking, for the coordinate information and the category information of the detected object, a plurality of objects in the images continuously captured by the camera, respectively, and assigning a fixed ID number to each detected object. The target tracking model may be, but is not limited to, an existing optical flow estimation or deepsorts tracking model, which facilitates subsequent collision prediction time calculations.
In another embodiment, in step S3, the process of determining whether there is an object in the expansion mode in the current image includes:
in step S31, an object that exists in both the current image and the history frame image is identified and noted as a first object. Specifically, object tracking may be performed according to the object tracking model in the object tracking step, so as to identify an object existing in the current image and the history frame image, and the history frame image is preferably but not limited to an image acquired by a camera in a previous frame or a preset previous period. In order to improve the accuracy of the collision prediction time, preferably, the historical frame image is the last frame image acquired by the camera.
Step S32, calculating the sizes of the first object in the current image and the history frame image, respectively. The size of the first object in the current image/history frame image is specifically the size of the image of the first object in the current image/history frame image. The size of the first object in the current image/history frame image is preferably, but not limited to, the area or width or height of the image of the first object at the image plane. Further preferably, in order to increase the speed of acquiring the size of the first object in the image plane, the size of a bounding box (rectangular prediction box) of the object detection model (e.g., YOLO or SSD) is taken as the size of the first object in the image plane. The bounding box is the smallest circumscribed rectangular prediction box of the first object, and its size is preferably, but not limited to, the area or width or height of the rectangular prediction box.
In step S33, if the size of the first object in the current image is greater than the size of the first object in the history frame image, the first object is considered to be in the expansion mode.
In this embodiment, in order to recognize the collision risk as early as possible, it is preferable that after detecting the object in the expansion mode, the object in the expansion mode is assigned a priority level according to the historical size change rate of the object, and the higher the historical size change rate of the object is, the higher the priority level is assigned, and the historical size change rate is specifically calculated by dividing the size of the object in the expansion mode in the previous frame image by the size of the object in the previous frame image, and when calculating the collision prediction time of the object in the expansion mode, the calculation is performed in order of the priority level from high to low. Further preferably, in order to quickly make collision avoidance response, after the collision prediction time of each object in the expansion mode is obtained by calculation, the collision prediction time is compared with a preset time threshold, and if the collision prediction time is less than or equal to the time threshold, a collision early warning signal is sent.
In the present embodiment, the collision prediction time of the object may be calculated by the following method:
the dimensional expansion speed of the object in the expansion mode is calculated,wherein S is p Representing the size of an object in the expanded mode in the history frame image, S c Representing the size of the object in the inflated mode in the current image; acquiring the acquisition frequency of a camera in advanceExtracting a mapping table corresponding to the acquisition frequency of the camera from the established mapping table library, and marking the mapping table as a target mapping table; and searching corresponding collision prediction time from the target mapping table based on the size expansion speed. Different camera acquisition frequencies correspond to different mapping tables, and the mapping tables are corresponding relation tables of the size expansion speed and the collision prediction time, which are established according to the multiple experimental results at the corresponding camera acquisition frequencies.
In another embodiment, in step S3, the process of calculating the collision prediction time of the object in the expansion mode is:
step A, calculating the size expansion speed:wherein S is p Representing the size of an object in the expanded mode in the history frame image, S c Representing the size of the object in the inflated mode in the current image.
Step B, calculating collision prediction time:wherein delta is t Representing the acquisition time difference between the historical frame image and the current image. When the history frame image is the previous frame image, Δ t Representing the time interval between acquisition of images by the camera.
The invention also discloses a computer readable storage medium storing a computer program, characterized in that in one embodiment, the computer program is executed by a processor to implement a collision prediction time calculation method as always provided in the above embodiment of the invention.
The invention also discloses an electronic device, as shown in fig. 3, the electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform a collision prediction time calculation method provided in the above-described embodiment of the present invention.
Fig. 3 is a schematic structural diagram of an electronic device for implementing a method for calculating a collision prediction time according to an embodiment of the present invention. The electronic device may comprise a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may further comprise a computer program stored in the memory 11 and executable on the processor 10, such as a collision prediction time calculation method program.
The processor 10 may be formed by an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be formed by a plurality of integrated circuits packaged with the same function or different functions, including one or more central processing units (Central Processing Unit, CPU), a microprocessor, a digital processing chip, a combination of a graphics processor and various control chips, etc. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects respective components of the entire electronic device using various interfaces and lines, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory 11 (for example, executing a collision prediction time calculation method program or the like), and calling data stored in the memory 11.
The memory 11 includes at least one type of readable storage medium including flash memory, a removable hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. The memory 11 may in other embodiments also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the electronic device. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device. The memory 11 may be used not only for storing application software installed in the electronic device and various types of data, such as codes of a collision prediction time calculation method program, but also for temporarily storing data that has been output or is to be output.
The communication bus 12 may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others. The bus may be classified as an address bus, a data bus, a control bus, etc. The bus is arranged to enable a connection communication between the memory 11 and the at least one processor 10 etc.
The communication interface 13 is used for communication between the above-described electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and/or a wireless interface (e.g., WI-FI interface, bluetooth interface, etc.), typically used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a Display (Display), an input unit such as a Keyboard (Keyboard), or alternatively a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, or the like. The display may also be referred to as a display screen or display unit, as appropriate, for displaying information processed in the electronic device and for displaying a visual user interface.
Fig. 3 shows only an electronic device with components, and it will be understood by those skilled in the art that the structure shown in fig. 3 is not limiting of the electronic device and may include fewer or more components than shown, or may combine certain components, or a different arrangement of components.
For example, although not shown, the electronic device may further include a power source (such as a battery) for powering the respective components, and the power source may be logically connected to the at least one processor 10 through a power management device, so as to perform functions of charge management, discharge management, and power consumption management through the power management device. The power supply may also include one or more of any of a direct current or alternating current power supply, recharging device, power failure detection circuit, power converter or inverter, power status indicator, etc. The electronic device may also include various sensors, bluetooth modules, wi-Fi modules, etc., which are not described in detail herein.
It should be understood that the examples are for illustrative purposes only and are not limited to this configuration in the scope of the patent application.
Further, the integrated modules/units of the electronic device 1 may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as separate products. The computer readable storage medium may be volatile or nonvolatile. For example, the computer readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM).
The invention also discloses a distributed visual collision sensing system, in an embodiment, as shown in fig. 4, the system further comprises: the cameras are arranged outside the protected main body in a linear array or an area array manner; the plurality of edge processing units respectively correspond to the cameras, respectively acquire output images of the corresponding cameras and execute the steps of the collision prediction time calculation method provided by the embodiment of the invention to acquire the collision prediction time of each camera detection area, and the edge processing units send the acquired collision prediction time to the central processing unit; and the central processing unit is used for receiving the collision prediction time sent by all the edge processing units, and outputting a collision alarm signal if at least one collision prediction time is smaller than or equal to a time threshold value.
In this embodiment, the protected subject is preferably, but not limited to, a vehicle, robot, or stationary object.
In this embodiment, the plurality of wide-angle cameras are distributed in a linear or planar array, wherein each camera in the distributed camera array is only responsible for sensing the view of the camera
The risk of collision (i.e. the detection area), each camera is insensitive to the collision objects that are farther from the collision point. Fig. 5 shows a schematic diagram of the cameras when the cameras are distributed outside the protected main body, a plurality of wide-angle cameras are arranged in a transverse or longitudinal linear array mode, and the number of the cameras can be flexibly determined according to a specific coverage angle. The camera 1a is responsible for perceiving the obstacle a in front and the camera 1B is responsible for perceiving the obstacle B in front thereof.
In this embodiment, fig. 6 discloses a schematic diagram of a distributed arrangement of camera area arrays. A plurality of wide-angle cameras are arranged in a surrounding area array mode, and the number of the cameras can be flexibly determined according to a specific coverage angle of view. As shown in fig. 6, the wide-angle cameras are arranged around the protected body in a circle, and each camera is only responsible for sensing the collision risk in the self visual field range (i.e. the detection area), if the collision risk is detected somewhere around the protected body (when the collision prediction time of some objects in the detection area of the camera at the place is less than the preset time threshold), the collision risk of the protected body at any position can be queried according to the camera installation distribution diagram.
In this embodiment, the edge processing unit is preferably, but not limited to, a microprocessor, such as an ARM, FPGA, or the like. The central processing unit is preferably, but not limited to, a server or a computer host or a microprocessor.
In the present embodiment, preferably, the central processing unit specifically performs:
and a, obtaining detection areas of the cameras according to the installation positions of the cameras outside the protected main body, and distributing detection area numbers for the detection areas of each camera.
Step b, the collision prediction time of the detection areas of all cameras is formed into an array, a schematic diagram of the array is shown in fig. 7, the collision prediction time of the detection areas of the cameras is represented by PU, each camera can have 0 or more collision prediction time, and the camera has 0 collision prediction time to represent that no expansion mode object exists in the current image of the camera.
And c, associating all collision prediction time of the detection area of the camera with the detection area number.
And d, outputting a collision alarm signal and a detection area number associated with the collision prediction time when any collision prediction time in the array is less than or equal to a time threshold. When all of the collision prediction times in the array are greater than the time threshold, then no collision risk is deemed to be present. The time threshold may be set empirically in advance according to the application field.
In this embodiment, preferably, as shown in fig. 4, the system further includes a communication module and an acousto-optic early warning prompt module, where the central processing unit outputs a collision alarm signal to the acousto-optic early warning prompt module through the communication module, and the acousto-optic early warning prompt module receives the collision alarm signal and then performs an acousto-optic alarm. The communication module is preferably, but not limited to, a radio frequency wireless communication module.
In this embodiment, as shown in fig. 4, the system further preferably further includes an equipment terminal early warning prompt module, and the central processing unit is connected to communicate with the equipment terminal early warning prompt module through a communication module. The device terminal early warning prompt module preferably but not limited to comprises a receiving unit, a processing unit and a loudspeaker unit of the terminal. The receiving unit is used for receiving collision alarm signals sent by the central processing unit through the communication module. The processing unit is used for monitoring whether the receiving unit receives the collision alarm signal or not, and starting the horn unit to send out alarm sound after receiving the collision alarm signal.
In an application scenario of the distributed visual collision sensing system provided by the present invention, fig. 8 discloses a specific flow schematic diagram of the scenario, which specifically includes:
step one: the plurality of wide-angle cameras are distributed and arranged in a linear array or an area array mode, wherein each camera in the distributed camera array is only responsible for sensing collision risks in the field of view of the camera, and each camera is insensitive to collision objects with far collision points.
Step two: the manner of distributed arrangement for a linear array is shown in fig. 5. A plurality of wide-angle cameras are arranged in a transverse or longitudinal linear array mode, and the number of the cameras can be flexibly determined according to a specific coverage angle of view. Wherein, as shown in fig. 5, the left camera 1a is only responsible for sensing the front obstacle a, the right camera 1B is only responsible for sensing the front obstacle B, and the rest of the camera detection does not perform collision risk detection.
Step three: the manner of distributed arrangement for an area array is shown in fig. 6. A plurality of wide-angle cameras are arranged in a surrounding area array mode, and the number of the cameras can be flexibly determined according to a specific coverage angle of view. As shown in fig. 6, the wide-angle cameras are arranged around the protected body in a circle, and each camera is only responsible for sensing collision risk in the field of view of the camera, and if the collision risk is detected at a certain position around the protected body, the collision risk at the certain position of the protected body can be queried according to the camera installation distribution diagram.
Step four: the current image acquired according to the installation mode of the second step or the third step is input to an edge processing unit (microchip) corresponding to each camera for image data processing, and an object in the field angle of the camera is detected by using a target detection model, so that corresponding detection information such as object coordinates, categories and the like is acquired. The object detection model may use, but is not limited to (SSD, YOL0, etc. detection models), and the detected object class includes, but is not limited to (human, animal, motor vehicle, non-motor vehicle, static object).
Step five: and constructing a tracking module. And (3) tracking a plurality of objects in the view angle of each camera by using the detected coordinate information and the category information respectively for the objects detected in the step (IV), and allocating a fixed ID number for each detected object. Wherein the target tracking model may use, but is not limited to, an optical flow estimation tracking model) or a deepsort tracking model.
Step six: and fifthly, storing information (coordinates, types, areas and the like) of the tracked object in the image in the step five through a data structure of a queue, judging whether the object has a far-small near-large expansion mode according to multi-frame historical image data, if the object has the expansion mode, calculating the collision time, otherwise, not performing subsequent calculation.
Step seven: after the object has an expansion mode, the Time To Collision (TTC) can be estimated according to the expansion speed of the object in the image. Wherein, the perspective principle schematic diagram of the far small and near large object in the image is shown in fig. 2. The Time To Collision (TTC) is calculated as follows:
dimensional expansion speed:
wherein S is 2 Representing the area or width or height of an object in the current image, S 1 Representing the area or width or height of the object in the last frame of image.
The collision prediction time of the object in the expansion mode is:
wherein delta is t For the time interval between acquisition of the current image and the previous frame of image.
Step eight: and the results obtained by processing and calculating by each edge processing unit are connected in series to construct a central processing unit array signal as shown in fig. 7. Further, whether the array signal is smaller than a set time threshold is judged, and if the array signal is smaller than the set time threshold, the collision risk is judged. And otherwise, judging that the collision risk exists, and simultaneously sending collision risk information to the wireless communication module. The time threshold can be flexibly set according to the application field.
Step nine: the wireless communication module receives the collision risk information and sends the collision risk information to the equipment terminal early warning prompt module.
Step ten: the acousto-optic early warning prompt module converts the collision risk information sent by the wireless communication module into corresponding acousto-optic warning, and then provides collision perception early warning prompt for the protected main body. Thus, the whole distributed visual collision sensing is completed.
According to the distributed vision collision sensing system provided by the invention, a plurality of cameras are arranged through a linear array or an area array, the collision risk of the surrounding environment is judged according to the calculated collision prediction time (TTC) through target detection, target tracking and collision prediction time calculation, compared with a radar scheme, the distributed vision collision sensing system has the advantages of strong anti-interference performance, low cost and simple development, the types of collision objects can be distinguished, the collision sensing accuracy is improved, and the false detection rate of collision sensing is reduced. Meanwhile, according to the method of calculating the collision prediction time (TTC) according to the object expansion speed, depth information of a collision object is not required to be acquired from a two-dimensional image through a deep learning algorithm or high-performance hardware, collision risk can be perceived without distance measurement, the collision perception stability is improved, and the consumption of system resources and the requirements of hardware performance are reduced. In addition, the camera installation and arrangement mode is flexible, simple and convenient, the application scene is wide, and the method is suitable for various scenes needing to be perceived as collision, including but not limited to (vehicles, robots and the like).
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the invention, the scope of which is defined by the claims and their equivalents.
Claims (10)
1. A collision prediction time calculation method, characterized by comprising:
step S1, acquiring a current image shot by a camera;
step S2, identifying an object in the current image based on a pre-trained target detection model;
step S3, judging whether an object in an expansion mode exists in the current image: if not, returning to the execution step S1; if so, calculating collision prediction time according to the size expansion speed of the object in the expansion mode in the current image and the historical frame image, and executing step S4;
and S4, outputting collision prediction time of all the objects in the expansion mode in the current image, and returning to the step S1.
2. The collision prediction time calculation method according to claim 1, wherein the process of judging whether or not there is an object in the expansion mode in the current image in step S3 includes:
identifying an object which exists in the current image and the historical frame image at the same time, and marking the object as a first object;
calculating the sizes of the first object in the current image and the historical frame image respectively;
and if the size of the first object in the current image is larger than the size of the first object in the historical frame image, the first object is considered to be in the expansion mode.
3. The method of claim 2, wherein the historical frame image is a previous frame image captured by a camera.
4. A collision predicted time calculation method according to any one of claims 1 to 3, wherein in step S3, the process of calculating the collision predicted time of the object in the expansion mode is:
calculating the size expansion speed:wherein S is p Representing the size of an object in the expanded mode in the history frame image, S c Representing the size of the object in the inflated mode in the current image;
calculating collision prediction time:wherein delta is t Representing the acquisition time difference between the historical frame image and the current image.
5. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements a collision prediction time calculation method according to any one of claims 1 to 4.
6. An electronic device, the electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform a collision prediction time calculation method according to any one of claims 1 to 4.
7. A distributed visual impact awareness system, comprising:
the cameras are arranged outside the protected main body in a linear array or an area array manner;
a plurality of edge processing units respectively corresponding to the cameras, respectively acquiring output images of the corresponding cameras and performing the steps of a collision prediction time calculation method according to any one of claims 1 to 4 to obtain a collision prediction time for each camera detection area, the edge processing units transmitting the obtained collision prediction time to the central processing unit;
and the central processing unit is used for receiving the collision prediction time sent by all the edge processing units, and outputting a collision alarm signal if at least one collision prediction time is smaller than or equal to a time threshold value.
8. The distributed vision impact sensing system of claim 7, wherein the central processor performs in particular:
obtaining detection areas of the cameras according to the installation positions of the cameras outside the protected main body, and distributing detection area numbers for the detection areas of each camera;
the collision prediction time of the detection areas of all cameras is formed into an array;
associating the collision prediction time of the detection area of the camera with the detection area number;
and outputting a detection area number associated with the collision alarm signal and the collision prediction time when any collision prediction time in the array is less than or equal to a time threshold.
9. The distributed vision collision sensing system of claim 7 or 8, further comprising a communication module and an audible and visual early warning prompt module, wherein the central processing unit outputs a collision warning signal to the audible and visual early warning prompt module through the communication module, and the audible and visual early warning prompt module performs audible and visual warning after receiving the collision warning signal.
10. The distributed vision collision sensing system of claim 9, further comprising a device terminal pre-warning prompt module, the central processing unit in communication with the device terminal pre-warning prompt module via the communication module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310849733.1A CN116863445A (en) | 2023-07-11 | 2023-07-11 | Collision prediction time calculation method and distributed vision collision sensing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310849733.1A CN116863445A (en) | 2023-07-11 | 2023-07-11 | Collision prediction time calculation method and distributed vision collision sensing system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116863445A true CN116863445A (en) | 2023-10-10 |
Family
ID=88223074
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310849733.1A Pending CN116863445A (en) | 2023-07-11 | 2023-07-11 | Collision prediction time calculation method and distributed vision collision sensing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116863445A (en) |
-
2023
- 2023-07-11 CN CN202310849733.1A patent/CN116863445A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3229041B1 (en) | Object detection using radar and vision defined image detection zone | |
| CN112417967B (en) | Obstacle detection method, obstacle detection device, computer device, and storage medium | |
| CN110084895B (en) | Method and equipment for marking point cloud data | |
| CN106952303B (en) | Vehicle distance detection method, device and system | |
| JP4173902B2 (en) | Vehicle periphery monitoring device | |
| EP3881226A1 (en) | Object classification using extra-regional context | |
| CN111814752B (en) | Indoor positioning realization method, server, intelligent mobile device and storage medium | |
| CN111753609A (en) | Target identification method and device and camera | |
| CN110442120B (en) | Method for controlling robot to move in different scenes, robot and terminal equipment | |
| KR20160137247A (en) | Apparatus and method for providing guidance information using crosswalk recognition result | |
| CN109747530A (en) | A kind of dual camera and millimeter wave merge automobile sensory perceptual system | |
| US11092690B1 (en) | Predicting lidar data using machine learning | |
| CN108021899A (en) | Vehicle intelligent front truck anti-collision early warning method based on binocular camera | |
| CN114495064A (en) | Monocular depth estimation-based vehicle surrounding obstacle early warning method | |
| CN111913177A (en) | Method and device for detecting target object and storage medium | |
| Liu et al. | Vehicle detection and ranging using two different focal length cameras | |
| JP2021033510A (en) | Driving assistance device | |
| CN116778448A (en) | Vehicle safe driving assistance method, device, system, equipment and storage medium | |
| CN116486351A (en) | Driving early warning method, device, equipment and storage medium | |
| CN114384486A (en) | Data processing method and device | |
| CN116863445A (en) | Collision prediction time calculation method and distributed vision collision sensing system | |
| US20220318456A1 (en) | Simulation method based on three-dimensional contour, storage medium, computer equipment | |
| CN114638947A (en) | Data labeling method and device, electronic equipment and storage medium | |
| CN104931024A (en) | Obstacle detection device | |
| US11164041B2 (en) | Semi-supervised learning with infrastructure |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |