CN114332776B - Non-motor vehicle occupant pedestrian lane detection method, system, device and storage medium - Google Patents
Non-motor vehicle occupant pedestrian lane detection method, system, device and storage medium Download PDFInfo
- Publication number
- CN114332776B CN114332776B CN202210215712.XA CN202210215712A CN114332776B CN 114332776 B CN114332776 B CN 114332776B CN 202210215712 A CN202210215712 A CN 202210215712A CN 114332776 B CN114332776 B CN 114332776B
- Authority
- CN
- China
- Prior art keywords
- motor vehicle
- target frame
- frame
- target
- pedestrian
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention provides a method, a system, equipment and a storage medium for detecting a pedestrian passageway of a non-motor vehicle, belonging to the technical field of image processing. The system comprises an acquisition module, a detection module, a construction module, an analysis module and a data transmission module; the acquisition module, the detection module, the construction module, the analysis module and the data transmission module are sequentially connected; the acquisition module is used for acquiring monitoring scene information to obtain a detection area; the detection module is used for acquiring target frames of the non-motor vehicles and the pedestrians, matching the non-motor vehicles and the pedestrians and acquiring a minimum adjacent rectangular target frame; the construction module is used for constructing a target image sequence; the analysis module is used for analyzing the state of the non-motor vehicle and predicting the running state of the non-motor vehicle; the data transmission module is used for carrying out data structuring processing on the prediction result and transmitting the message information and any one piece of picture data in the target image sequence to the cloud. The problems of time and labor waste and high cost of manual inspection are solved.
Description
Technical Field
The application relates to a detection method, in particular to a method, a system, equipment and a storage medium for detecting a pedestrian passageway occupied by a non-motor vehicle, and belongs to the technical field of image processing.
Background
With the rapid development of the economy of China and the continuous improvement of the living standard of people, all lanes are gradually occupied by motor lanes on the road, and some cities are provided with non-motor lanes. Even in Shenzhen, where the public transportation system is relatively developed, we can often see non-motor vehicles traveling on sidewalks. The core lifting and liner hanging device has the disadvantages of poor safety and comfort when pedestrians on the sidewalk walk; the non-motor vehicles are difficult to drive, and the efficiency of a traffic system is low; even serious traffic accidents occur, which endangers the life safety of people and causes serious economic property loss. Therefore, it becomes important how to accurately identify the illegal behavior that the non-motor vehicle occupies the sidewalk in the monitoring scene, which is helpful to feed back the more road sections of the non-motor vehicle to relevant departments in time, so as to perform targeted road improvement, further improve the image of the city, and improve the safety and comfort of people going out.
In the traditional detection work of a non-motor vehicle driving on a sidewalk, professionals such as a traffic police are usually arranged to carry out regular inspection, but the method is low in efficiency, consumes great manpower and material resources, and meanwhile, the detection effect is not comprehensive.
In another method, non-motor vehicle detection is performed using a Support Vector Machine (SVM) and a HOG feature (Histogram of Oriented Gradient) in the field of machine learning, and analysis of the driving state of a non-motor vehicle is performed using a moving object detection algorithm such as an optical flow method or a background subtraction method. On one hand, the method needs to consume a great deal of time and energy on feature extraction, and on the other hand, the accuracy on SVM detection is not high enough. The detection algorithm of moving objects by using an optical flow method and the like is also very complicated, the test on an algorithm designer is large, and the identification effect of the method is poor.
Therefore, a detection method, a system, an electronic device and a storage medium for detecting the occupation of the sidewalk by the non-motor vehicle in driving, which can solve the problems of time and labor waste and high cost of the traditional manual inspection, can realize the intelligent and real-time effective monitoring of the road and provide an effective decision basis for a management layer.
Disclosure of Invention
The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. It should be understood that this summary is not an exhaustive overview of the invention. It is not intended to determine the key or critical elements of the present invention, nor is it intended to limit the scope of the present invention. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is discussed later.
In view of the above, in order to solve the technical problems of time and labor waste and high cost in the prior art, the invention provides a method, a system, equipment and a storage medium for detecting a pedestrian path occupied by a non-motor vehicle.
The first scheme is as follows: the detection system for the pedestrian passageway occupied by the non-motor vehicle comprises an acquisition module, a detection module, a construction module, an analysis module and a data transmission module; the acquisition module, the detection module, the construction module, the analysis module and the data transmission module are sequentially connected; the acquisition module is used for acquiring monitoring scene information to obtain a detection area; the detection module is used for acquiring target frames of the non-motor vehicles and the pedestrians, matching the non-motor vehicles and the pedestrians and acquiring a minimum adjacent rectangular target frame; the construction module is used for constructing a target image sequence; the analysis module is used for analyzing the state of the non-motor vehicle and predicting the running state of the non-motor vehicle; the data transmission module is used for carrying out data structuring processing on the prediction result, outputting message information and transmitting the message information and any one piece of picture data in the target image sequence to the cloud.
Scheme II: the detection method for the non-motor vehicle occupying the sidewalk comprises the following steps:
acquiring scene information under monitoring, and acquiring a detection area based on the scene information;
secondly, acquiring target frames of the non-motor vehicles and the pedestrians based on the detection area, and matching the non-motor vehicles and the pedestrians to obtain a minimum adjacent rectangular target frame;
thirdly, constructing a target image sequence based on the minimum adjacent matrix in the second step and the position of the detection area in the first step;
analyzing the small adjacent matrix area by using the target image sequence in the step three to obtain the state of the non-motor vehicle;
and step five, predicting the running state of the non-motor vehicle according to the state of the non-motor vehicle in the step four.
Preferably, the specific method for acquiring scene information under monitoring based on the scene information in the first step includes the following steps:
accessing real-time video stream information acquired by a camera;
decoding an original video into a single-frame picture in a uniform RGB format;
step three, converting a color space and carrying out image filtering and denoising treatment on a single-frame picture;
identifying sidewalks in a scene and positioning the sidewalks at a pixel level through a semantic segmentation algorithm, and then generating a minimum external rotation matrix capable of completely framing sidewalk areas according to segmentation results, wherein the minimum external rotation matrix is a detection area;
and step one and five, generating a unique label after the detection area is obtained.
Preferably, in the second step, the specific method for obtaining the target frames of the non-motor vehicles and the pedestrians based on the detection area and matching the non-motor vehicles and the pedestrians to obtain the minimum adjacent rectangular target frame includes the following steps:
step two, acquiring a target frame of the non-motor vehicle by utilizing a target detection algorithm yolov3 based on deep learning for each frame of the monitored image, acquiring the coordinates of the central point of the target frame of the non-motor vehicle, and generating a unique label for the target frame of the non-motor vehicle;
secondly, acquiring a pedestrian target frame for each frame of the monitored image by using a target detection algorithm yolov3 based on deep learning, calculating the coordinates of the central point of the pedestrian target frame, and generating a unique label for the pedestrian target frame;
step two, calculating the distance between the center point of the pedestrian target frame and the center point of the non-motor vehicle target frame for each non-motor vehicle target frame, and drawing a circle by taking the distance as a radius and taking the center point of the non-motor vehicle target frame as an origin;
step two, selecting pedestrian target frames with the radius smaller than a set threshold, wherein the number of the pedestrian target frames is not more than 3;
step two, calculating the proportion of the area of the pedestrian target frame to the total area of the circle in the circle overlapping area, and taking the pedestrian target frame with the largest occupation ratio; if the number of the pedestrian target frames is more than the maximum number, taking the pedestrian target frame with the smallest circle radius; the method specifically comprises the following steps:
step two, fifthly, establishing a new reference system, taking the central point of the non-motor vehicle target frame as an origin, and projecting the coordinates of the monitoring area into the reference system:
determining the value range of the random points;
step two, step three, randomly generating a plurality of points, counting the number of the points in the rectangular area, and counting the percentage;
step two, framing the pedestrian target frame and the non-motor vehicle target frame in the step two by using a minimum adjacent rectangle to form a minimum adjacent rectangle target frame, and calculating the center point coordinate of the minimum adjacent rectangle; generating unique labels for the minimum adjacent rectangular target frame, the pedestrian target frame and the non-motor vehicle target frame;
and step two, comparing the coordinates of the center point of the minimum adjacent rectangular target frame with the coordinates of the detection area, judging whether the center point of the minimum adjacent rectangular target frame is in the detection area, if not, stopping tracking the minimum adjacent rectangular target frame, and if so, performing the next step.
Preferably, when the non-motor vehicle and the person are matched in the step two, if the non-motor vehicle target frame cannot be matched with the pedestrian target frame, no operation is performed on the non-motor vehicle target frame; if the distance is less than 1 pedestrian target frame of a certain set threshold, then the minimum adjacent rectangle is directly generated for the pedestrian target frame and the non-motor vehicle target frame.
Preferably, the specific method for constructing the target image sequence in the third step is that the label of the minimum adjacent rectangular target frame in each frame of target monitoring image and the minimum adjacent rectangular target frame in each frame of target monitoring image are sequentially extracted, and the target frames are scaled to the same size to construct the image sequence for the non-motor vehicle and pedestrian targets to be identified; if the length of the image sequence exceeds 20 frames, 20 frames are randomly extracted in time sequence.
Preferably, the specific method for obtaining the state of the non-motor vehicle in the step four comprises the following steps:
step four, selecting a fixed point in the target image sequence as a reference point, wherein the fixed point is selected at a certain corner of the edge of the detection area;
step two, calculating a vector from a fixed point of each frame of the target image sequence to the midpoint of the minimum adjacent rectangular target frame in the step two, then calculating a slope, and sequentially storing the slopes corresponding to the target image sequence into an array;
step three, judging the state of the non-motor vehicle according to the slope trend and the change magnitude, and specifically comprising the following steps:
(1) if the range of the slope array is smaller than a certain set threshold value, the non-motor vehicle is in a non-motion state, and the non-motor vehicle is judged not to be in a running state;
(2) if the range of the slope array is larger than a certain set threshold value, the non-motor vehicle is in a motion state, and the non-motor vehicle is judged to be in a running state; calculating the difference of adjacent slopes of the array, wherein the slope corresponding to the (i + 1) th frame-the slope corresponding to the ith frame form a new difference array in sequence according to the difference;
(3) if the ith difference is smaller than 0, the non-motor vehicle drives to the area where the fixed point is located from the ith frame to the i +1 frame, otherwise, the non-motor vehicle drives away from the area where the fixed point is located.
Preferably, the concrete method for predicting the driving state of the non-motor vehicle in the step five comprises the following steps:
fifthly, inputting the target image sequence into a CNN (content-based network) feature extraction network, and extracting the features of each frame of image in the target image sequence, wherein each frame of target monitoring image shares the same CNN network;
step two, the spatial characteristics of each frame of image are transformed into a data form acceptable by an LSTM time sequence modeling network; each LSTM unit receives the spatial characteristics output by a frame of CNN network as input, and the output of the last LSTM unit outputs a group of cell states after internal processing, and the relevance of the non-motor vehicle characteristics on the time sequence is constructed once;
fifthly, splicing the cell states output by each LSTM, and inputting the cell states into a full-connection FC driving state analysis network to determine a final state; the FC running state analysis network output layer is provided with two neurons which respectively represent pushing and riding, if the neurons representing riding are activated and the score is higher than a set threshold value t, the output result is riding, otherwise, pushing is output;
step five four, when the number of the riding frames exceeds a preset frame number threshold value, the non-motor vehicle is in a riding state; and when the pushed frame number exceeds a preset frame number threshold value, the non-motor vehicle is in a pushing state.
The third scheme is as follows: an electronic device comprising a memory storing a computer program and a processor implementing the steps of the method for detecting the occupation of a sidewalk by a non-motor vehicle when the processor executes the computer program.
And the scheme is as follows: a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out a method of detecting that a non-motor vehicle occupies a sidewalk.
The invention has the following beneficial effects: the method can effectively judge the condition that the non-motor vehicle occupies the sidewalk, solves the problems of time and labor waste and high cost of the traditional manual inspection, realizes the intelligent and real-time effective monitoring of the road, and provides an effective decision basis for a management layer.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a schematic diagram of a system architecture;
FIG. 2 is a schematic process flow diagram;
FIG. 3 is a schematic view of a non-motor vehicle target frame and a pedestrian target frame not being superimposed;
FIG. 4 is a schematic view of the coincidence of a non-motor vehicle target frame and a pedestrian target frame;
FIG. 5 is a schematic view of a minimum contiguous rectangular target box;
fig. 6 is a schematic diagram of a fixed point with a coordinate system established.
Detailed Description
In order to make the technical solutions and advantages of the embodiments of the present application more apparent, the following further detailed description of the exemplary embodiments of the present application with reference to the accompanying drawings makes it clear that the described embodiments are only a part of the embodiments of the present application, and are not exhaustive of all embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
Embodiment 1, the present embodiment is described with reference to fig. 1, and a detection system for a pedestrian crossing of a non-motor vehicle includes an acquisition module, a detection module, a construction module, an analysis module, and a data transmission module; the acquisition module, the detection module, the construction module, the analysis module and the data transmission module are sequentially connected; the acquisition module is used for acquiring monitoring scene information to obtain a detection area; the detection module is used for acquiring target frames of the non-motor vehicles and the pedestrians, matching the non-motor vehicles and the pedestrians and acquiring a minimum adjacent rectangular target frame; the construction module is used for constructing a target image sequence; the analysis module is used for analyzing the state of the non-motor vehicle and predicting the running state of the non-motor vehicle; the data transmission module is used for carrying out data structuring processing on the prediction result, outputting message information and transmitting the message information and any one piece of picture data in the target image sequence to the cloud.
Specifically, if the output result is "riding", message information is formed. The message information should include at least the date and time of day, the serial number of the non-motor vehicle object, the code of the edge computing gateway device, and location information.
The message data and the picture data are sent to the cloud end through the post function of the https, the edge computing gateway serves as a https client end to send the data, and the cloud end serves as an https server to receive and store the information. If the edge computing gateway fails to send, a cache mechanism is adopted to store the data which fails to send locally and resend the data at other time.
Example 2, the present embodiment is described with reference to fig. 2 to 6, and the method for detecting a pedestrian path occupied by a non-motor vehicle includes the following steps:
acquiring scene information under monitoring, and acquiring a detection area based on the scene information; the method comprises the following steps:
accessing real-time video stream information acquired by a camera; specifically, the RJ45 ethernet cable is used to connect the camera with the network interface of the edge computing gateway, and the edge computing gateway is in the system software of the edge computing gateway and accesses the real-time video stream information collected by the camera in the RTSP video stream address mode.
Decoding an original video into a single-frame picture in a uniform RGB format; and the picture is improved, and the detection area and the further processing of the subsequent modules are conveniently obtained.
Step three, converting a color space and carrying out image filtering and denoising treatment on a single-frame picture;
identifying sidewalks in a scene and positioning the sidewalks at a pixel level through a semantic segmentation algorithm, and then generating a minimum external rotation matrix capable of completely framing sidewalk areas according to segmentation results, wherein the minimum external rotation matrix is a detection area;
and step one and five, generating a unique label after the detection area is obtained.
The specific detection area can be manually selected, and can also be determined by using algorithm self-adaptive detection analysis; for algorithm self-adaptive detection analysis, a sidewalk in a scene can be identified and positioned at a pixel level through a semantic segmentation algorithm, and then a minimum external rotation matrix capable of completely framing a sidewalk area, namely an identification frame of the sidewalk, is generated according to a segmentation result and is used as a detection area. Since the camera is generally fixed and the position of the sidewalk is difficult to change, the detection area can be determined only at the beginning of monitoring, and the detection area can be used all the time.
Secondly, acquiring target frames of the non-motor vehicles and the pedestrians based on the detection area, and matching the non-motor vehicles and the pedestrians to obtain a minimum adjacent rectangular target frame; the method comprises the following steps:
step two, acquiring a target frame of the non-motor vehicle by utilizing a target detection algorithm yolov3 based on deep learning for each frame of the monitored image, acquiring the coordinates of the central point of the target frame of the non-motor vehicle, and generating a unique label for the target frame of the non-motor vehicle;
secondly, acquiring a pedestrian target frame for each frame of the monitored image by using a target detection algorithm yolov3 based on deep learning, calculating the coordinates of the central point of the pedestrian target frame, and generating a unique label for the pedestrian target frame;
step two, for each non-motor vehicle target frame, calculating the distance between the center point of the pedestrian target frame and the center point of the non-motor vehicle target frame, and drawing a circle by taking the distance as a radius and taking the center point of the non-motor vehicle target frame as an origin;
step two, taking pedestrian target frames with the radius smaller than a set threshold, wherein the number of the pedestrian target frames is not more than 3;
step two, calculating the proportion of the area of the pedestrian target frame to the total area of the circle in the circle overlapping area, and taking the pedestrian target frame with the largest occupation ratio; if the number of the pedestrian target frames is more than the maximum number, taking the pedestrian target frame with the smallest circle radius; the method specifically comprises the following steps:
step two, fifthly, establishing a new reference system, taking the central point of the non-motor vehicle target frame as an origin, and projecting the coordinates of the monitoring area into the reference system:
determining the value range of the random point;
step two, step three, randomly generating a plurality of points, counting the number of the points in the rectangular area, and counting the percentage;
step two, framing the pedestrian target frame and the non-motor vehicle target frame in the step two by using a minimum adjacent rectangle to form a minimum adjacent rectangle target frame, and calculating the center point coordinate of the minimum adjacent rectangle; generating unique labels for the minimum adjacent rectangular target frame, the pedestrian target frame and the non-motor vehicle target frame;
and step two, comparing the coordinates of the center point of the minimum adjacent rectangular target frame with the coordinates of the detection area, judging whether the center point of the minimum adjacent rectangular target frame is in the detection area, if not, stopping tracking the minimum adjacent rectangular target frame, and if so, performing the next step.
Specifically, when the non-motor vehicle is matched with the person, if the non-motor vehicle target frame cannot be matched with the pedestrian target frame, no operation is performed on the non-motor vehicle target frame; if the distance is less than 1 pedestrian target frame of a certain set threshold, then the minimum adjacent rectangle is directly generated for the pedestrian target frame and the non-motor vehicle target frame.
Specifically, the same adjacent rectangle in the continuous frame target detection images is associated by using a target tracking algorithm, and a unique target sequence number is allocated to each adjacent rectangle until the adjacent rectangle target frame disappears or the adjacent rectangle leaves the detection area.
Specifically, the definition of the same adjacent rectangle is: two adjacent rectangles are considered to be the same adjacent rectangle if they have the same reference numeral.
Specifically, if the non-motor vehicle re-enters the detection area, a new target serial number should be assigned. The target serial number can be formed by randomly combining 8 or more digits or letters, and at least each new target serial number is unique in the current day.
Specifically, the minimum adjacent rectangle can reflect the maximum range of the space area occupied by the minimum adjacent rectangle and has stronger robustness to the tiny change of the target frame; the central point can also represent the relative central position of the two, and the judgment of whether the two are in the detection area is most reasonable by using the central point as a basis.
Step three, constructing a target image sequence based on the minimum adjacent matrix in the step two and the position of the detection area in the step one, wherein the specific method comprises the following steps: sequentially extracting the label of the minimum adjacent rectangular target frame in each frame of target monitoring image and the minimum adjacent rectangular target frame of each frame of target monitoring image, zooming the target frames to the same size, and constructing an image sequence for the non-motor vehicle and pedestrian targets to be identified; if the length of the image sequence exceeds 20 frames, 20 frames are randomly extracted in time sequence.
Step four, analyzing the small adjacent matrix area by using the target image sequence in the step three to obtain the state of the non-motor vehicle, wherein the specific method comprises the following steps:
fourthly, the camera is fixed, so that a fixed point is selected from the target image sequence as a reference point, and the fixed point is selected from a certain corner of the edge of the detection area; referring to fig. 6, in the present embodiment, the detection box is selected to be in the upper left corner, and a new coordinate system is established for the origin.
Step two, calculating a vector from a fixed point of each frame of the target image sequence to the midpoint of the minimum adjacent rectangular target frame in the step two, then calculating a slope, and sequentially storing the slopes corresponding to the target image sequence into an array; if the slope is just infinite, it is considered as an extra large value.
Step three, judging the state of the non-motor vehicle according to the slope trend and the change magnitude, and specifically comprising the following steps:
(1) if the range of the slope array is smaller than a certain set threshold value, the non-motor vehicle is in a non-motion state, and the non-motor vehicle is judged not to be in a running state;
(2) if the range of the slope array is larger than a certain set threshold value, the non-motor vehicle is in a motion state, and the non-motor vehicle is judged to be in a running state; calculating the difference of adjacent slopes of the array, wherein the slope corresponding to the (i + 1) th frame-the slope corresponding to the ith frame form a new difference array in sequence according to the difference;
(3) if the ith difference is smaller than 0, the non-motor vehicle drives to the area where the fixed point is located from the ith frame to the i +1 frame, otherwise, the non-motor vehicle drives away from the area where the fixed point is located.
Referring to fig. 6, the center point of the minimum bounding rectangular object box is indicated in the figure from frame I to frame II to frame III, and the moving direction is indicated by the dashed arrow. And calculating the vector and the slope from the origin to the central point. And judging the difference value between the slope corresponding to the (i + 1) th frame and the slope corresponding to the ith frame. It is clear that in this figure the slope decreases gradually and therefore the difference is negative and therefore considered to move towards the area where the fixed point is located, which coincides with the actual direction of motion.
The deviation condition of the central point of the target frame can be amplified by utilizing the slope for judgment, so that the motion condition can be judged better. Meanwhile, a certain corner of the detection area is selected as an origin, which is beneficial to avoiding the situation that the center point is deviated but the slope is almost unchanged.
Step five, predicting the running state of the non-motor vehicle according to the state of the non-motor vehicle in the step four;
specifically, the driving state of the non-motor vehicle is a continuous behavior in time, and in order to acquire characteristics in time and space at the same time and output an accurate classification result, the target sequence is input into a preset CNN-LSTM neural network for analysis. The method comprises the following steps:
fifthly, inputting the target image sequence into a CNN (content-based network) feature extraction network, and extracting the features of each frame of image in the target image sequence, wherein each frame of target monitoring image shares the same CNN network;
step two, the spatial characteristics of each frame of image are transformed into a data form acceptable by an LSTM time sequence modeling network; each LSTM unit receives the spatial characteristics output by a frame of CNN network as input, and the output of the last LSTM unit outputs a group of cell states after internal processing, and the relevance of the non-motor vehicle characteristics on the time sequence is constructed once;
specifically, the LSTM units are neural units in an LSTM neural network.
Fifthly, splicing the cell states output by each LSTM, and inputting the cell states into a full-connection FC driving state analysis network to determine a final state; the FC running state analysis network output layer is provided with two neurons which respectively represent pushing and riding, if the neurons representing riding are activated and the score is higher than a set threshold value t, the output result is riding, otherwise, pushing is output;
step five four, when the number of the riding frames exceeds a preset frame number threshold value, the non-motor vehicle is in a riding state; and when the pushed frame number exceeds a preset frame number threshold value, the non-motor vehicle is in a pushing state.
The computer device of the present invention may be a device including a processor, a memory, and the like, for example, a single chip microcomputer including a central processing unit and the like. And the processor is used for implementing the steps of the recommendation method capable of modifying the relationship-driven recommendation data based on the CREO software when executing the computer program stored in the memory.
The Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. In addition, the memory may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
Computer-readable storage medium embodiments
The computer readable storage medium of the present invention may be any form of storage medium that can be read by a processor of a computer device, including but not limited to non-volatile memory, ferroelectric memory, etc., and the computer readable storage medium has stored thereon a computer program that, when the computer program stored in the memory is read and executed by the processor of the computer device, can implement the above-mentioned steps of the CREO-based software that can modify the modeling method of the relationship-driven modeling data.
The computer program comprises computer program code which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as described herein. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. The present invention has been disclosed in an illustrative rather than a restrictive sense, and the scope of the present invention is defined by the appended claims.
Claims (9)
1. The detection system for the pedestrian passageway occupied by the non-motor vehicle is characterized by comprising an acquisition module, a detection module, a construction module, an analysis module and a data transmission module; the acquisition module, the detection module, the construction module, the analysis module and the data transmission module are sequentially connected; the acquisition module is used for acquiring monitoring scene information to obtain a detection area; the detection module is used for acquiring target frames of the non-motor vehicles and the pedestrians, matching the non-motor vehicles and the pedestrians and acquiring a minimum adjacent rectangular target frame, and comprises the following steps:
step two, acquiring a target frame of the non-motor vehicle by utilizing a target detection algorithm yolov3 based on deep learning for each frame of the monitored image, acquiring the coordinates of the central point of the target frame of the non-motor vehicle, and generating a unique label for the target frame of the non-motor vehicle;
secondly, acquiring a pedestrian target frame for each frame of the monitored image by using a target detection algorithm yolov3 based on deep learning, calculating the coordinates of the central point of the pedestrian target frame, and generating a unique label for the pedestrian target frame;
step two, calculating the distance between the center point of the pedestrian target frame and the center point of the non-motor vehicle target frame for each non-motor vehicle target frame, and drawing a circle by taking the distance as a radius and taking the center point of the non-motor vehicle target frame as an origin;
step two, taking pedestrian target frames with the radius smaller than a set threshold, wherein the number of the pedestrian target frames is not more than 3;
step two, calculating the proportion of the area of the pedestrian target frame to the total area of the circle in the circle overlapping area, and taking the pedestrian target frame with the largest occupation ratio; if the number of the pedestrian target frames is larger than the maximum number of the pedestrian target frames, selecting the pedestrian target frame with the smallest circle radius; the method specifically comprises the following steps:
step two, fifthly, establishing a new reference system, taking the central point of the non-motor vehicle target frame as an origin, and projecting the coordinates of the monitoring area into the reference system:
determining the value range of the random point;
step two, step three, randomly generating a plurality of points, counting the number of the points in the rectangular area, and counting the percentage;
step two, framing the pedestrian target frame and the non-motor vehicle target frame in the step two by using a minimum adjacent rectangle to form a minimum adjacent rectangle target frame, and calculating the center point coordinate of the minimum adjacent rectangle; generating unique labels for the minimum adjacent rectangular target frame, the pedestrian target frame and the non-motor vehicle target frame;
comparing the coordinates of the center point of the minimum adjacent rectangular target frame with the coordinates of the detection area, judging whether the center point of the minimum adjacent rectangular target frame is in the detection area, if not, stopping tracking the minimum adjacent rectangular target frame, and if so, performing the next step;
the construction module is used for constructing a target image sequence; the analysis module is used for analyzing the state of the non-motor vehicle and predicting the running state of the non-motor vehicle; the data transmission module is used for carrying out data structuring processing on the prediction result, outputting message information and transmitting the message information and any one piece of picture data in the target image sequence to the cloud.
2. The detection method for the non-motor vehicle occupying the sidewalk is characterized by comprising the following steps:
acquiring scene information under monitoring, and acquiring a detection area based on the scene information;
secondly, acquiring target frames of the non-motor vehicles and the pedestrians based on the detection area, and matching the non-motor vehicles and the pedestrians to obtain a minimum adjacent rectangular target frame; the method comprises the following steps:
step two, acquiring a target frame of the non-motor vehicle by utilizing a target detection algorithm yolov3 based on deep learning for each frame of the monitored image, acquiring the coordinates of the central point of the target frame of the non-motor vehicle, and generating a unique label for the target frame of the non-motor vehicle;
secondly, acquiring a pedestrian target frame for each frame of the monitored image by using a target detection algorithm yolov3 based on deep learning, calculating the coordinates of the central point of the pedestrian target frame, and generating a unique label for the pedestrian target frame;
step two, calculating the distance between the center point of the pedestrian target frame and the center point of the non-motor vehicle target frame for each non-motor vehicle target frame, and drawing a circle by taking the distance as a radius and taking the center point of the non-motor vehicle target frame as an origin;
step two, selecting pedestrian target frames with the radius smaller than a set threshold, wherein the number of the pedestrian target frames is not more than 3;
step two, calculating the proportion of the area of the pedestrian target frame to the total area of the circle in the circle overlapping area, and taking the pedestrian target frame with the largest occupation ratio; if the number of the pedestrian target frames is more than the maximum number, taking the pedestrian target frame with the smallest circle radius; the method specifically comprises the following steps:
step two, fifthly, establishing a new reference system, taking the central point of the non-motor vehicle target frame as an origin, and projecting the coordinates of the monitoring area into the reference system:
determining the value range of the random point;
step two, step three, randomly generating a plurality of points, counting the number of the points in the rectangular area, and counting the percentage;
step two, framing the pedestrian target frame and the non-motor vehicle target frame in the step two by using a minimum adjacent rectangle to form a minimum adjacent rectangle target frame, and calculating the center point coordinate of the minimum adjacent rectangle; generating unique labels for the minimum adjacent rectangular target frame, the pedestrian target frame and the non-motor vehicle target frame;
comparing the coordinates of the central point of the minimum adjacent rectangular target frame with the coordinates of the detection area, judging whether the central point of the minimum adjacent rectangular target frame is in the detection area, if not, stopping tracking the minimum adjacent rectangular target frame, and if so, performing the next step;
thirdly, constructing a target image sequence based on the minimum adjacent rectangle in the second step and the position of the detection area in the first step;
analyzing the small adjacent matrix area by using the target image sequence in the step three to obtain the state of the non-motor vehicle;
and step five, predicting the running state of the non-motor vehicle according to the state of the non-motor vehicle in the step four.
3. The method for detecting the pedestrian walkway occupied by the non-motor vehicle according to claim 2, wherein the step one of acquiring scene information under monitoring comprises the following steps of:
accessing real-time video stream information acquired by a camera;
decoding an original video into a single-frame picture in a unified RGB format;
step three, converting a color space and carrying out image filtering and denoising treatment on a single-frame picture;
identifying sidewalks in a scene and positioning the sidewalks at a pixel level through a semantic segmentation algorithm, and then generating a minimum external rotation matrix capable of completely framing sidewalk areas according to segmentation results, wherein the minimum external rotation matrix is a detection area;
and step one and five, generating a unique label after the detection area is obtained.
4. The method for detecting the pedestrian walkway occupied by the non-motor vehicle according to claim 3, wherein when the non-motor vehicle and the person are matched in the step two, if the non-motor vehicle target frame cannot be matched with the pedestrian target frame, no operation is performed on the non-motor vehicle target frame; if the distance is less than 1 pedestrian target frame of a certain set threshold, then the minimum adjacent rectangle is directly generated for the pedestrian target frame and the non-motor vehicle target frame.
5. The method for detecting the pedestrian walkway occupied by the non-motor vehicle according to claim 4, wherein the specific method for constructing the target image sequence in the third step is to sequentially extract the label of the minimum adjacent rectangular target frame in each frame of the target monitoring image and the minimum adjacent rectangular target frame in each frame of the target monitoring image, and scale the target frames to the same size to construct the image sequence for the non-motor vehicle and the pedestrian target to be identified; if the length of the image sequence exceeds 20 frames, 20 frames are randomly extracted in time sequence.
6. The method for detecting the pedestrian walkway occupied by the non-motor vehicle according to claim 5, wherein the concrete method for obtaining the state of the non-motor vehicle in the fourth step is that the method comprises the following steps:
step four, selecting a fixed point in the target image sequence as a reference point, wherein the fixed point is selected at a certain corner of the edge of the detection area;
step two, calculating a vector from a fixed point of each frame of the target image sequence to the midpoint of the minimum adjacent rectangular target frame in the step two, then calculating a slope, and sequentially storing the slopes corresponding to the target image sequence into an array;
step three, judging the state of the non-motor vehicle according to the slope trend and the change magnitude, and specifically comprising the following steps:
(1) if the range of the slope array is smaller than a certain set threshold value, the non-motor vehicle is in a non-motion state, and the non-motor vehicle is judged not to be in a running state;
(2) if the range of the slope array is larger than a certain set threshold value, the non-motor vehicle is in a motion state, and the non-motor vehicle is judged to be in a running state; calculating the difference of adjacent slopes of the array, wherein the slope corresponding to the (i + 1) th frame-the slope corresponding to the ith frame form a new difference array according to the difference of the adjacent slopes of the array in sequence;
(3) if the ith difference is smaller than 0, the non-motor vehicle drives to the area where the fixed point is located from the ith frame to the i +1 frame, otherwise, the non-motor vehicle drives away from the area where the fixed point is located.
7. The method for detecting the pedestrian crosswalk occupied by the non-motor vehicle according to claim 6, wherein the concrete method for predicting the driving state of the non-motor vehicle in the step five is that the method comprises the following steps:
fifthly, inputting the target image sequence into a CNN (content-based network) feature extraction network, and extracting the features of each frame of image in the target image sequence, wherein each frame of target monitoring image shares the same CNN network;
step two, the spatial characteristics of each frame of image are transformed into a data form acceptable by an LSTM time sequence modeling network; each LSTM unit receives the spatial characteristics output by a frame of CNN network as input, and the output of the last LSTM unit outputs a group of cell states after internal processing, and the relevance of the non-motor vehicle characteristics on the time sequence is constructed once;
fifthly, splicing the cell states output by each LSTM, and inputting the cell states into a full-connection FC driving state analysis network to determine a final state; the FC running state analysis network output layer is provided with two neurons which respectively represent pushing and riding, if the neurons representing riding are activated and the score is higher than a set threshold value t, the output result is riding, otherwise, pushing is output;
step five four, when the number of the riding frames exceeds a preset frame number threshold value, the non-motor vehicle is in a riding state; and when the pushed frame number exceeds a preset frame number threshold value, the non-motor vehicle is in a pushing state.
8. An electronic device, comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of the method for detecting a pedestrian walkway occupied by a non-motor vehicle according to any one of claims 2 to 7 when executing the computer program.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method for detecting the occupation of a sidewalk by a non-motor vehicle according to any one of claims 2 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210215712.XA CN114332776B (en) | 2022-03-07 | 2022-03-07 | Non-motor vehicle occupant pedestrian lane detection method, system, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210215712.XA CN114332776B (en) | 2022-03-07 | 2022-03-07 | Non-motor vehicle occupant pedestrian lane detection method, system, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114332776A CN114332776A (en) | 2022-04-12 |
| CN114332776B true CN114332776B (en) | 2022-08-02 |
Family
ID=81030429
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210215712.XA Active CN114332776B (en) | 2022-03-07 | 2022-03-07 | Non-motor vehicle occupant pedestrian lane detection method, system, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114332776B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114898342B (en) * | 2022-07-15 | 2022-11-25 | 深圳市城市交通规划设计研究中心股份有限公司 | Method for detecting call receiving and making of non-motor vehicle driver in driving |
| CN115083208B (en) * | 2022-07-20 | 2023-02-03 | 深圳市城市交通规划设计研究中心股份有限公司 | Human-vehicle conflict early warning method, early warning analysis method, electronic device and storage medium |
| CN115512315B (en) * | 2022-11-01 | 2023-04-18 | 深圳市城市交通规划设计研究中心股份有限公司 | Non-motor vehicle child riding detection method, electronic equipment and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021051868A1 (en) * | 2019-09-20 | 2021-03-25 | 平安科技（深圳）有限公司 | Target location method and apparatus, computer device, computer storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105956568A (en) * | 2016-05-11 | 2016-09-21 | 东华大学 | Abnormal behavior detecting and early warning method based on monitored object identification |
| CN111898475A (en) * | 2020-07-10 | 2020-11-06 | 浙江大华技术股份有限公司 | Method and device for estimating state of non-motor vehicle, storage medium, and electronic device |
| CN112163543B (en) * | 2020-10-12 | 2024-03-26 | 北京易华录信息技术股份有限公司 | Detection method and system for illegal lane occupation of vehicle |
| CN112597945A (en) * | 2020-12-29 | 2021-04-02 | 上海眼控科技股份有限公司 | Vehicle detection method and device |
-
2022
- 2022-03-07 CN CN202210215712.XA patent/CN114332776B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021051868A1 (en) * | 2019-09-20 | 2021-03-25 | 平安科技（深圳）有限公司 | Target location method and apparatus, computer device, computer storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114332776A (en) | 2022-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114332776B (en) | Non-motor vehicle occupant pedestrian lane detection method, system, device and storage medium | |
| WO2020173022A1 (en) | Vehicle violation identifying method, server and storage medium | |
| EP3806064A1 (en) | Method and apparatus for detecting parking space usage condition, electronic device, and storage medium | |
| CN109740420B (en) | Vehicle law violation identification method and related product | |
| Slimani et al. | An automated license plate detection and recognition system based on wavelet decomposition and CNN | |
| CN110348463B (en) | Method and device for identifying vehicle | |
| CN113160575A (en) | Traffic violation detection method and system for non-motor vehicles and drivers | |
| CN104978567A (en) | Vehicle detection method based on scenario classification | |
| CN114898342B (en) | Method for detecting call receiving and making of non-motor vehicle driver in driving | |
| Xiang et al. | Lightweight fully convolutional network for license plate detection | |
| Azad et al. | New method for optimization of license plate recognition system with use of edge detection and connected component | |
| CN112651293B (en) | Video detection method for road illegal spreading event | |
| CN112749653A (en) | Pedestrian detection method, device, electronic equipment and storage medium | |
| Darapaneni et al. | Computer vision based license plate detection for automated vehicle parking management system | |
| Parvin et al. | Vehicle number plate detection and recognition techniques: a review | |
| Yépez et al. | Deep learning‐based embedded license plate localisation system | |
| CN115512315B (en) | Non-motor vehicle child riding detection method, electronic equipment and storage medium | |
| Zhang et al. | Image-based approach for parking-spot detection with occlusion handling | |
| Rokonuzzaman et al. | Automatic vehicle identification system using machine learning and robot operating system (ROS) | |
| Ciuntu et al. | Real-time traffic sign detection and classification using machine learning and optical character recognition | |
| CN116597270A (en) | Road damage target detection method based on attention mechanism integrated learning network | |
| CN115909140A (en) | Video target segmentation method and system based on high-order video monitoring | |
| CN115294774A (en) | Non-motor vehicle road illegal parking detection method and device based on deep learning | |
| CN113378845A (en) | Scene segmentation method, device, equipment and storage medium | |
| Bassam et al. | Smart parking system based on improved OCR model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |