CN111582300A - High-dynamic target detection method based on event camera - Google Patents
High-dynamic target detection method based on event camera Download PDFInfo
- Publication number
- CN111582300A CN111582300A CN202010198845.1A CN202010198845A CN111582300A CN 111582300 A CN111582300 A CN 111582300A CN 202010198845 A CN202010198845 A CN 202010198845A CN 111582300 A CN111582300 A CN 111582300A
- Authority
- CN
- China
- Prior art keywords
- event
- image
- camera
- prediction
- high dynamic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a high dynamic target detection method based on an event camera, which comprises the steps of obtaining event stream data from the event camera and converting the event stream into an event image; adjusting the size of the event image, and converting the event image into a uniform resolution; putting the event image into a convolutional neural network to obtain a plurality of prediction frames; and obtaining a detection result with higher confidence through non-maximum value inhibition. The invention improves the capability of high dynamic target detection against illumination interference and complex background environment interference by combining the event camera with deep learning.
Description
Technical Field
The invention belongs to the technical field of computer vision, and particularly relates to a high-dynamic target detection method based on an event camera.
Background
Object detection is a computer vision technique that finds the location and size of an object in an image. The technology can be applied to the fields of artificial intelligence systems, vehicle driving assistance systems, target behavior analysis, intelligent video monitoring and the like.
An event camera, unlike a normal camera, is a biologically inspired visual sensor that outputs changes in pixel level brightness. The event camera behaves as a traditional vision sensor running at speeds of thousands of frames per second, but the amount of data is much smaller. The event camera has low overall power, small data storage, low computational performance requirements, and an increased dynamic range of several orders of magnitude over conventional cameras. Such cameras are less susceptible to motion blur and can provide reliable visual information during high speed motion.
The current high dynamic moving object detection method mainly uses a common camera. These methods are susceptible to illumination, and when the light intensity is weak, it is difficult to detect a moving object. In addition, the detection effect is much worse when the background is very complicated or the moving speed of the moving object is fast and the image blur occurs.
Disclosure of Invention
In order to solve the problems that the existing moving target detection method is easily influenced by external illumination, is easily influenced by a complex background environment and is easily influenced by motion blur, the invention provides a high-dynamic target detection method based on an event camera, and the specific technical scheme of the invention is as follows:
a high dynamic target detection method based on an event camera is characterized by comprising the following steps:
s1: obtaining event stream data from an event camera and converting the event stream into an event image;
s2: adjusting the size of the event image, and converting the event image into a uniform resolution;
s3: putting the event image into a convolutional neural network to obtain a plurality of prediction frames;
s4: and obtaining a detection result with higher confidence through non-maximum value inhibition.
Further, the step S1 includes the following steps:
s11: acquiring event stream data from an event camera;
the data representation mode of the event camera is address event expression, and each data is composed of an event address, namely the position of the event on the image, the time stamp of the event and the polarity of the event;
s12: generating an event image;
after acquiring the event stream data from the event camera, events within a fixed time interval △ t are accumulated to generate an event image, which is defined as
Wherein AE isframeFor an event image, tstartIs a starting time, tendTo the end time, tevIs the current time, AExyFor a triggered event, time interval △ t is tend-tstart＝10ms；
S13: removing noise;
setting a threshold value N, and for an event image, if the number of events around an event is less than the threshold value N, then the event is taken as noise to be removed, and for each event, the method is adopted for processing, and then the event image after noise removal is formed.
Further, the step S2 converts the obtained event image into an image with a resolution of 416 × 416.
Further, the convolutional neural network of step S3 is tiny yolo, and step S3 includes the following steps:
s31: extracting 13 x 13 and 26 x 26 feature maps through the convolutional and pooling layers;
s32: and predicting the characteristic diagram to obtain x, y, w, h and confidence of the target, wherein coordinates (x, y) represent the relative values of the center of the frame and the boundary of the grid unit, w and h are the width and the height of the frame, and the confidence is the IOU of the predicted boundary frame and the real boundary frame, wherein the IOU refers to the intersection-union ratio, namely the ratio of the intersection to the union.
Further, the step S4 first finds out the prediction box with the highest confidence from the prediction boxes obtained in the step S3, then calculates the IOU between the prediction box and the rest prediction boxes, and if the value of the IOU is greater than 0.4, the prediction box is removed; the above process is then repeated for the remaining prediction blocks until all prediction blocks have been processed.
The invention has the beneficial effects that:
1. the invention improves the capability of high dynamic target detection against illumination interference and complex background environment interference by combining the event camera with deep learning.
2. Compared with the prior art, the method has small data volume, the data volume of the event camera is much smaller than that of the traditional camera, and the required storage space is less than that of the common camera; the power consumption is low, the power consumption of the event camera is low, the selected neural network model is small, the occupied computing resources are few, and the power consumption is further reduced.
3. The method of the invention resists strong light and weak light, the dynamic range of the event camera is larger than that of the common camera, when the illumination intensity is particularly large or small, the event camera can still work normally, and a high dynamic target is detected; the method is resistant to complex background interference, the event camera senses illumination intensity change, namely when no object moves in the camera industry, the camera cannot output, when the moving target is detected, the camera can only shoot a moving person or other moving objects, no matter how complex the background is, the complex background cannot be shot, and therefore the detection of the moving target by the event camera cannot be influenced.
4. The method of the invention can detect the moving object which moves rapidly: processing the raw data of the event camera can generate event images with high frame rate, and the event images are not easy to generate motion blur, so that the motion images can be well detected even if the motion speed of a moving object is high.
Drawings
In order to illustrate embodiments of the present invention or technical solutions in the prior art more clearly, the drawings which are needed in the embodiments will be briefly described below, so that the features and advantages of the present invention can be understood more clearly by referring to the drawings, which are schematic and should not be construed as limiting the present invention in any way, and for a person skilled in the art, other drawings can be obtained on the basis of these drawings without any inventive effort. Wherein:
FIG. 1 is a flow chart of a method for high dynamic object detection based on an event camera according to the present invention;
FIG. 2 is an event image of the present invention;
FIG. 3 is a network architecture diagram of the present invention;
FIG. 4 is a graph showing the detection effect of the method of the present invention.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings. It should be noted that the embodiments of the present invention and features of the embodiments may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.
As shown in fig. 1, a high dynamic target detection method based on an event camera includes the following steps:
s1: obtaining event stream data from an event camera and converting the event stream into an event image;
s2: adjusting the size of the event image, and converting the event image into a uniform resolution;
s3: putting the event image into a convolutional neural network to obtain a plurality of prediction frames;
s4: and obtaining a detection result with higher confidence through non-maximum value inhibition.
Step S1 includes the following steps:
s11: acquiring event stream data from an event camera;
the data representation mode of the event camera is address event expression, and each data is composed of an event address, namely the position of the event on the image, the time stamp of the event and the polarity of the event;
s12: generating an event image;
after acquiring the event stream data from the event camera, events within a fixed time interval △ t are accumulated to generate an event image, which is defined as
Wherein AE isframeFor an event image, tstartIs a starting time, tendTo the end time, tevIs the current time, AExyFor a triggered event, time interval △ t is tend-tstart＝10ms；
S13: removing noise;
setting a threshold value N, and for an event image, if the number of events around an event is less than the threshold value N, then the event is taken as noise to be removed, and for each event, the method is adopted for processing, and then the event image after noise removal is formed.
Step S2 converts the obtained event image into an image with a resolution of 416 × 416, so as to facilitate the processing of the neural network.
The convolutional neural network of step S3 is tiny yolo, the structure is shown in fig. 3, and step S3 includes the following steps:
s31: extracting 13 x 13 and 26 x 26 feature maps through the convolutional and pooling layers;
s32: and predicting the characteristic diagram to obtain x, y, w, h and confidence of the target, wherein coordinates (x, y) represent the relative values of the center of the frame and the boundary of the grid unit, w and h are the width and the height of the frame, and the confidence is the IOU of the predicted boundary frame and the real boundary frame, wherein the IOU refers to the intersection-union ratio, namely the ratio of the intersection to the union.
In the step S4, first, a prediction box with the highest confidence is found from the prediction boxes obtained in the step S3, and then the IOUs between the prediction box and the rest of the prediction boxes are calculated, and if the value of the IOU is greater than 0.4, the prediction box is removed; the above process is then repeated for the remaining prediction blocks until all prediction blocks have been processed.
Fig. 4 is a detection effect diagram of the method of the present invention, and it can be seen from the diagram that the method of the present invention can detect the category to which the moving object captured by the event camera belongs and the position in the image, and the generated event image has no redundant background information except for the moving object, and is not easily interfered by the complex background. Through test, the method can achieve good detection effect when the target moves rapidly and the illumination is weak.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (5)
1. A high dynamic target detection method based on an event camera is characterized by comprising the following steps:
s1: obtaining event stream data from an event camera and converting the event stream into an event image;
s2: adjusting the size of the event image, and converting the event image into a uniform resolution;
s3: putting the event image into a convolutional neural network to obtain a plurality of prediction frames;
s4: and obtaining a detection result with higher confidence through non-maximum value inhibition.
2. The method for detecting high dynamic objects based on event camera as claimed in claim 1, wherein said step S1 includes the following steps:
s11: acquiring event stream data from an event camera;
the data representation mode of the event camera is address event expression, and each data is composed of an event address, namely the position of the event on the image, the time stamp of the event and the polarity of the event;
s12: generating an event image;
after acquiring event stream data from the event camera, accumulating events within a fixed time interval delta t to generate an event image, wherein the event image is defined as
Wherein AE isframeFor an event image, tstartIs a starting time, tendTo the end time, tevIs the current time, AExyFor a triggered event, the time interval Δ t is tend-tstart＝10ms；
S13: removing noise;
setting a threshold value N, and for an event image, if the number of events around an event is less than the threshold value N, then the event is taken as noise to be removed, and for each event, the method is adopted for processing, and then the event image after noise removal is formed.
3. The method for detecting the high dynamic objects based on the event camera as claimed in claim 1, wherein the step S2 is to convert the obtained event image into an image with a resolution of 416 x 416.
4. The method for detecting high dynamic objects based on event camera as claimed in claim 1, wherein the convolutional neural network of step S3 is tiny yolo, and the step S3 comprises the following steps:
s31: extracting 13 x 13 and 26 x 26 feature maps through the convolutional and pooling layers;
s32: and predicting the characteristic diagram to obtain x, y, w, h and confidence of the target, wherein coordinates (x, y) represent the relative values of the center of the frame and the boundary of the grid unit, w and h are the width and the height of the frame, and the confidence is the IOU of the predicted boundary frame and the real boundary frame, wherein the IOU refers to the intersection-union ratio, namely the ratio of the intersection to the union.
5. The method for detecting high dynamic objects based on event camera as claimed in claim 1, wherein the step S4 first finds the prediction box with the highest confidence from the prediction boxes obtained in the step S3, then calculates the IOU between the prediction box and the rest of the prediction boxes, and if the value is greater than 0.4, then the prediction box is removed; the above process is then repeated for the remaining prediction blocks until all prediction blocks have been processed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010198845.1A CN111582300A (en) | 2020-03-20 | 2020-03-20 | High-dynamic target detection method based on event camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010198845.1A CN111582300A (en) | 2020-03-20 | 2020-03-20 | High-dynamic target detection method based on event camera |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111582300A true CN111582300A (en) | 2020-08-25 |
Family
ID=72122441
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010198845.1A Pending CN111582300A (en) | 2020-03-20 | 2020-03-20 | High-dynamic target detection method based on event camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111582300A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111931752A (en) * | 2020-10-13 | 2020-11-13 | 中航金城无人系统有限公司 | Dynamic target detection method based on event camera |
| CN112816995A (en) * | 2020-12-25 | 2021-05-18 | 北京灵汐科技有限公司 | Target detection method and device, fusion processing unit and computer readable medium |
| CN113160218A (en) * | 2021-05-12 | 2021-07-23 | 深圳龙岗智能视听研究院 | Method for detecting object motion intensity based on event camera |
| CN113159217A (en) * | 2021-05-12 | 2021-07-23 | 深圳龙岗智能视听研究院 | Attention mechanism target detection method based on event camera |
| CN113506229A (en) * | 2021-07-15 | 2021-10-15 | 清华大学 | Neural network training and image generation method and device |
| CN113537071A (en) * | 2021-07-19 | 2021-10-22 | 深圳龙岗智能视听研究院 | Static and dynamic target detection method and device based on event camera |
| CN113572998A (en) * | 2021-09-22 | 2021-10-29 | 中科南京智能技术研究院 | Data collection method and system based on event camera |
| CN113660455A (en) * | 2021-07-08 | 2021-11-16 | 深圳宇晰科技有限公司 | Method, system and terminal for fall detection based on DVS data |
| CN114037741A (en) * | 2021-10-12 | 2022-02-11 | 中国电子科技南湖研究院 | Adaptive target detection method and device based on event camera |
| WO2022237591A1 (en) * | 2021-05-08 | 2022-11-17 | 北京灵汐科技有限公司 | Moving object identification method and apparatus, electronic device, and readable storage medium |
| WO2023273952A1 (en) * | 2021-06-29 | 2023-01-05 | 华为技术有限公司 | Data processing method and apparatus and electronic device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108182670A (en) * | 2018-01-15 | 2018-06-19 | 清华大学 | A kind of resolution enhancement methods and system of event image |
| WO2019099337A1 (en) * | 2017-11-14 | 2019-05-23 | Kaban Technologies Llc | Event camera-based deformable object tracking |
| CN110399908A (en) * | 2019-07-04 | 2019-11-01 | 西北工业大学 | Classification method and device based on event mode camera, storage medium, electronic device |
| CN110599414A (en) * | 2019-08-28 | 2019-12-20 | 武汉大学 | Event camera data processing-oriented time-space normalization method |
-
2020
- 2020-03-20 CN CN202010198845.1A patent/CN111582300A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019099337A1 (en) * | 2017-11-14 | 2019-05-23 | Kaban Technologies Llc | Event camera-based deformable object tracking |
| CN108182670A (en) * | 2018-01-15 | 2018-06-19 | 清华大学 | A kind of resolution enhancement methods and system of event image |
| CN110399908A (en) * | 2019-07-04 | 2019-11-01 | 西北工业大学 | Classification method and device based on event mode camera, storage medium, electronic device |
| CN110599414A (en) * | 2019-08-28 | 2019-12-20 | 武汉大学 | Event camera data processing-oriented time-space normalization method |
Non-Patent Citations (1)
| Title |
|---|
| 张姝: "基于地址事件表示信号的动态手势识别方法", 《中国优秀硕士学位论文全文数据库》 * |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111931752A (en) * | 2020-10-13 | 2020-11-13 | 中航金城无人系统有限公司 | Dynamic target detection method based on event camera |
| CN112816995A (en) * | 2020-12-25 | 2021-05-18 | 北京灵汐科技有限公司 | Target detection method and device, fusion processing unit and computer readable medium |
| CN112816995B (en) * | 2020-12-25 | 2024-03-01 | 北京灵汐科技有限公司 | Target detection method and device, fusion processing unit and computer readable medium |
| WO2022237591A1 (en) * | 2021-05-08 | 2022-11-17 | 北京灵汐科技有限公司 | Moving object identification method and apparatus, electronic device, and readable storage medium |
| CN113160218A (en) * | 2021-05-12 | 2021-07-23 | 深圳龙岗智能视听研究院 | Method for detecting object motion intensity based on event camera |
| CN113159217A (en) * | 2021-05-12 | 2021-07-23 | 深圳龙岗智能视听研究院 | Attention mechanism target detection method based on event camera |
| CN113159217B (en) * | 2021-05-12 | 2023-08-01 | 深圳龙岗智能视听研究院 | Attention mechanism target detection method based on event camera |
| CN113160218B (en) * | 2021-05-12 | 2023-06-20 | 深圳龙岗智能视听研究院 | Method for detecting object motion intensity based on event camera |
| WO2023273952A1 (en) * | 2021-06-29 | 2023-01-05 | 华为技术有限公司 | Data processing method and apparatus and electronic device |
| CN113660455A (en) * | 2021-07-08 | 2021-11-16 | 深圳宇晰科技有限公司 | Method, system and terminal for fall detection based on DVS data |
| CN113506229A (en) * | 2021-07-15 | 2021-10-15 | 清华大学 | Neural network training and image generation method and device |
| CN113506229B (en) * | 2021-07-15 | 2024-04-12 | 清华大学 | Neural network training and image generating method and device |
| CN113537071B (en) * | 2021-07-19 | 2023-08-11 | 深圳龙岗智能视听研究院 | Static and dynamic target detection method and equipment based on event camera |
| CN113537071A (en) * | 2021-07-19 | 2021-10-22 | 深圳龙岗智能视听研究院 | Static and dynamic target detection method and device based on event camera |
| CN113572998A (en) * | 2021-09-22 | 2021-10-29 | 中科南京智能技术研究院 | Data collection method and system based on event camera |
| CN114037741A (en) * | 2021-10-12 | 2022-02-11 | 中国电子科技南湖研究院 | Adaptive target detection method and device based on event camera |
| CN114037741B (en) * | 2021-10-12 | 2023-09-19 | 中国电子科技南湖研究院 | Self-adaptive target detection method and device based on event camera |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111582300A (en) | High-dynamic target detection method based on event camera | |
| CN111931752B (en) | Dynamic target detection method based on event camera | |
| CN107872644B (en) | Video monitoring method and device | |
| WO2011028379A2 (en) | Foreground object tracking | |
| WO2011028380A2 (en) | Foreground object detection in a video surveillance system | |
| CN111260738A (en) | Multi-scale target tracking method based on relevant filtering and self-adaptive feature fusion | |
| CN105046719B (en) | A kind of video frequency monitoring method and system | |
| CN109377499B (en) | Pixel-level object segmentation method and device | |
| CN114885074B (en) | Event camera denoising method based on space-time density | |
| CN112232356A (en) | Event camera denoising method based on cluster degree and boundary characteristics | |
| CN112801027A (en) | Vehicle target detection method based on event camera | |
| CN114140656A (en) | Marine ship target identification method based on event camera | |
| CN110602504A (en) | Video decompression method and system based on YOLOv2 target detection algorithm | |
| CN111767826A (en) | Timing fixed-point scene abnormity detection method | |
| Angelo | A novel approach on object detection and tracking using adaptive background subtraction method | |
| CN112613564A (en) | Target detection post-processing method for eliminating overlapped frames | |
| CN111260686A (en) | Target tracking method and system for anti-shielding multi-feature fusion of self-adaptive cosine window | |
| CN108492308B (en) | Method and system for determining variable light split flow based on mutual structure guided filtering | |
| Qin et al. | A shadow removal algorithm for ViBe in HSV color space | |
| CN113066077B (en) | Flame detection method and device | |
| CN106780646B (en) | Parameter-free background modeling method suitable for multiple scenes | |
| Zhao et al. | An improved VIBE algorithm for fast suppression of ghosts and static objects | |
| CN113936030A (en) | Moving target detection method and system based on convolutional coding | |
| Gu et al. | Drms: Dim-light robust monocular simultaneous localization and mapping | |
| CN112668466A (en) | Lane line identification method for address event data stream |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200825 |
|
| RJ01 | Rejection of invention patent application after publication |