CN114255424A - Method and device for determining object behaviors, storage medium and electronic device - Google Patents
Method and device for determining object behaviors, storage medium and electronic device Download PDFInfo
- Publication number
- CN114255424A CN114255424A CN202111545195.4A CN202111545195A CN114255424A CN 114255424 A CN114255424 A CN 114255424A CN 202111545195 A CN202111545195 A CN 202111545195A CN 114255424 A CN114255424 A CN 114255424A
- Authority
- CN
- China
- Prior art keywords
- target
- determining
- behavior
- line
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Analysis (AREA)
Abstract
The embodiment of the invention provides a method, a device, a storage medium and an electronic device for determining object behaviors, wherein the method comprises the following steps: acquiring a target image obtained by shooting a target area; determining a warning line included in the target area based on the target image; determining a target moving track of a target object included in a target image based on an image sequence in which the target image is located; and determining the behavior of the target object based on the position relation between the warning line and the target moving track. By the method and the device, the problem that the behavior of the determined object is inaccurate in the related technology is solved, and the effect of improving the accuracy of the behavior of the determined object is achieved.
Description
Technical Field
The embodiment of the invention relates to the field of communication, in particular to a method and a device for determining object behaviors, a storage medium and an electronic device.
Background
With the continuous development of deep learning technology and network camera equipment, the automation and intellectualization of video monitoring technology gradually become the mainstream development trend of the current technology. Compared with the traditional video monitoring technology which can only carry out real-time monitoring and recording, the intelligent video monitoring utilizes the computer vision technology to process and understand the input video signals, realizes the automatic detection, tracking and analysis functions of interested targets and events in video scenes, and provides related early warning, thereby greatly reducing the labor cost of video analysis and improving the intelligent analysis efficiency. However, in the related art, there is a problem that the determination of the behavior of the object using the intelligent video monitoring is inaccurate.
In view of the above problems in the related art, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a method and a device for determining object behaviors, a storage medium and an electronic device, which are used for at least solving the problem of inaccurate object behavior determination in the related art.
According to an embodiment of the present invention, there is provided a method for determining an object behavior, including: acquiring a target image obtained by shooting a target area; determining a warning line included in the target area based on the target image; determining a target moving track of a target object included in the target image based on the image sequence in which the target image is located; and determining the behavior of the target object based on the position relation between the warning line and the target moving track.
According to another embodiment of the present invention, there is provided an apparatus for determining a behavior of an object, including: the acquisition module is used for acquiring a target image obtained by shooting a target area; a first determination module for determining a warning line included in the target area based on the target image; a second determining module, configured to determine a target movement trajectory of a target object included in the target image based on an image sequence in which the target image is located; and the third determination module is used for determining the behavior of the target object based on the position relation between the warning line and the target moving track.
According to yet another embodiment of the invention, there is also provided a computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by a processor, implements the steps of the method as set forth in any of the above.
According to yet another embodiment of the present invention, there is also provided an electronic device, including a memory in which a computer program is stored and a processor configured to execute the computer program to perform the steps in any of the above method embodiments.
According to the invention, the target image obtained by shooting the target area is obtained, the warning line included in the target area is determined according to the target image, the target moving track of the target object included in the target image is determined according to the image sequence of the target image, and the behavior of the target object is determined according to the position relation between the warning line and the target moving track. The behavior of the target object can be analyzed according to the position relation between the target moving track and the warning line, so that the position relation between the target object and the warning line can be accurately determined, and the behavior of the target object can be accurately determined.
Drawings
Fig. 1 is a block diagram of a hardware structure of a mobile terminal of a method for determining an object behavior according to an embodiment of the present invention;
FIG. 2 is a flow diagram of a method of determining object behavior according to an embodiment of the invention;
FIG. 3 is a schematic of a fence according to an exemplary embodiment of the present invention;
FIG. 4 is a first diagram illustrating the behavior of determining a target object according to an exemplary embodiment of the present invention;
FIG. 5 is a diagram of determining the behavior of a target object according to an exemplary embodiment of the present invention;
FIG. 6 is a flow diagram of a method for determining object behavior in accordance with a specific embodiment of the present invention;
fig. 7 is a block diagram of the structure of an apparatus for determining the behavior of an object according to an embodiment of the present invention.
Detailed Description
Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings in conjunction with the embodiments.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
The method embodiments provided in the embodiments of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. Taking an example of the application in a mobile terminal, fig. 1 is a block diagram of a hardware structure of the mobile terminal of a method for determining an object behavior according to an embodiment of the present invention. As shown in fig. 1, the mobile terminal may include one or more (only one shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data, wherein the mobile terminal may further include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration, and does not limit the structure of the mobile terminal. For example, the mobile terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
The memory 104 may be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the method for determining object behaviors in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, so as to implement the above-mentioned method. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the mobile terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
In this embodiment, a method for determining an object behavior is provided, and fig. 2 is a flowchart of the method for determining an object behavior according to the embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:
step S202, acquiring a target image obtained by shooting a target area;
step S204, determining a warning line included in the target area based on the target image;
step S206, determining a target moving track of a target object included in the target image based on the image sequence of the target image;
step S208, determining the behavior of the target object based on the position relation between the warning line and the target moving track.
In the above-described embodiment, the target area may be an area that can be captured by an image capturing apparatus that captures a target image. The image sequence may be images continuously captured by the imaging apparatus, the image sequence including a target image, and the warning line included in the target area and the target movement locus of the target object included in the target image may be determined from the image sequence. According to the position relation between the target moving track and the warning line, the position relation between the target object and the warning line can be determined, and therefore the behavior of the target object is determined.
In the above embodiments, the target object may be a vehicle, a person, an article, an animal, or the like. When the target object is a person, the target area may be an area in a traffic road. The target image may be an image taken by a camera installed above a road, at a traffic gate, or the like, and the warning line may be an edge line of the road. When a person crosses the road, real-time monitoring can be carried out, video recording is stored, and the target moving track of the target object is determined. When the target object is a vehicle, the guard line may be an edge line of a no-parking area or the like. When the target object is an animal, the warning line may be a margin line of an area where a stationary target object exits or enters. The target image of the target area can be continuously acquired through the camera equipment, the target moving track of the target object is determined according to the target image, and the behavior of the target object is determined according to the target moving track and the warning line. When the behavior of the target object does not meet the preset requirement, operations such as acousto-optic warning and the like can be carried out.
In the above embodiment, the warning line included in the target area may be automatically determined according to the target image without manually setting the warning area and the warning line. The efficiency of determining the behavior of the object can be improved by reducing the labor cost.
Optionally, the main body of the above steps may be a background processor, or other devices with similar processing capabilities, and may also be a machine integrated with at least an image acquisition device and a data processing device, where the image acquisition device may include a graphics acquisition module such as a camera, and the data processing device may include a terminal such as a computer and a mobile phone, but is not limited thereto.
According to the invention, the target image obtained by shooting the target area is obtained, the warning line included in the target area is determined according to the image sequence of the target image, the target moving track of the target object included in the target image is determined according to the target image, and the behavior of the target object is determined according to the position relation between the warning line and the target moving track. The behavior of the target object can be analyzed according to the position relation between the target moving track and the warning line, so that the position relation between the target object and the warning line can be accurately determined, and the behavior of the target object can be accurately determined.
In an exemplary embodiment, the determining the behavior of the target object based on the position relationship between the warning line and the target moving trajectory includes: determining a first distance between the end point of the target moving track and the warning line; determining a behavior of the target object based on the first distance. In this embodiment, when determining the behavior of the target object, a first distance between the end point of the target movement trajectory and the warning line may be determined, and the behavior of the target object may be determined according to the first distance. Wherein the first distance may be a first distance from the warning line within the warning region determined by the warning line.
In an exemplary embodiment, the determining a first distance between the end point of the target moving trajectory and the warning line includes: in response to the position relationship indicating that the target movement track and the warning line have an intersection, determining whether an end point of the target movement track is updated within a first predetermined time based on an image captured after the target image is captured; in response to the endpoint not being updated, determining a first distance of the endpoint from the warning line. In the present embodiment, in the case where the positional relationship indicates that there is an intersection of the target movement trajectory and the warning line, it may be determined whether or not the end point of the target movement trajectory is updated within the first predetermined time from the image captured after the target image is captured. In the event that an update has not occurred to the endpoint, a first distance from the endpoint to the warning line is determined. The first predetermined time may be a customized time, which is not limited in the present invention, for example, 5s, 10s, 20s, and the like.
In an exemplary embodiment, the alert line comprises a no-go alert line, and the determining the behavior of the target object based on the first distance comprises: in response to the first distance being greater than a predetermined distance, determining the behavior of the target object as being a passing alert zone behavior; or in response to the first distance being less than or equal to the predetermined distance, determining the behavior of the target object as out-of-range behavior. In this embodiment, in the case where the guard line is the no-entry guard line, the relationship between the first distance and the predetermined distance may be determined, and in the case where the first distance is greater than the predetermined distance, the behavior of the target object is determined as being a behavior passing through the guard area. And determining the behavior of the target object as the out-of-range behavior in the case that the first distance is smaller than or equal to the predetermined distance. That is, when the first distance is less than or equal to the predetermined distance, it may be considered that the target object stops after crossing the guard line, and the behavior of the target object is determined to be an out-of-range behavior.
In the above embodiment, in a case where the positional relationship indicates that there is no intersection between the target movement trajectory and the warning line, it is determined that the behavior of the target object is a non-boundary-crossing behavior.
In an exemplary embodiment, the alert line includes a stay-prohibited alert line, and the determining the behavior of the target object based on the first distance includes: in response to determining that the target object enters an alert zone determined based on the alert line based on the first distance, determining a time at which the target object enters the alert zone; and in response to the time being greater than a second predetermined time, determining that the behavior of the target object is an illegal stay behavior. In this embodiment, in the case that the alert line is the stay-prohibited alert line, the time when the target object enters the alert area may be determined according to the first distance, and in the case that the time when the target object enters is greater than the second predetermined time, the behavior of the target object may be determined to be the illegal stay behavior. The warning area is an area determined by a warning line, when the warning area is a parking prohibition area and the target object is a vehicle, if the first distance indicates that the target object is in the warning area, the time for the vehicle to enter the parking prohibition area can be determined, and if the time exceeds the second predetermined time, the vehicle can be determined to be parked in the warning area, so that the behavior of the vehicle can be determined to be an illegal parking behavior.
In one exemplary embodiment, the alert line includes a plurality of no-pass alert lines, and determining the behavior of the target object based on the positional relationship of the alert line to the target movement trajectory includes: and determining the behavior of the target object as a behavior of crossing an alert zone in response to the position relationship indicating that the target movement trajectory has an intersection with a plurality of alert lines included in the alert lines. In the present embodiment, in the case where the positional relationship indicates that the target movement locus has an intersection with each of the plurality of guard lines included in the guard line, it may be determined that the target object behaves so as to pass through the guard region, that is, the target object has passed through the guard region.
In the above embodiments, the guard line may include one or more guard lines. The schematic diagram of the warning line can be seen in fig. 3, and as shown in fig. 3, the solid line in the figure is the warning line. In a traffic road scene, one side edge of a road can be determined as a warning line, and two side edges of the road can also be determined as warning lines.
In an exemplary embodiment, the warning line is determined based on an edge line of the target zone. In this embodiment, the edge line of the target area may be directly determined as the warning line, or may be a line having a certain distance from the edge line. For example, the shift is performed based on the edge line to obtain the warning line.
In the above embodiment, the edge line of the target area of the target object may be determined, and the edge line may be determined as the warning line. For example, a sub-area of the target type included in the target area may be determined, and an edge line of the sub-area may be determined as an alert line. The target type may be a type corresponding to a scene of the target image. When the scene of the target image is a traffic scene, the target type may be a road type, and the sub-region of the target type may be a traffic road. When the scene of the target image is a parking scene, the target type may be a parking prohibition region type, and the sub-region of the target type is a parking prohibition region. When the scene of the target image is a farm, the target type can be a breeding type, and the sub-region of the target type is a breeding region. That is, the sub-region of the target type and the scene of the target image satisfy the preset corresponding relationship, the sub-region of the target type included in the target region may be determined according to the corresponding relationship, and then the warning line is determined based on the edge line of the sub-region.
In one exemplary embodiment, before determining the warning line included in the target area based on the target image, the method further includes: detecting a target scene included in the target image, and generating an edge line of the target area based on the target scene; determining the warning line based on the margin line. In this embodiment, before determining the warning line, a target scene included in the target image may be detected, and the edge line of the target area may be generated from the target scene. The target scene may include a traffic road scene, a parking scene, and the like.
In one exemplary embodiment, detecting a target scene included in the target image and generating an edge line of the target region based on the target scene includes: determining the target scene included in the target image based on the trained target network model, and determining the edge line of the target area based on the target scene, wherein the target network model is trained in the following manner: acquiring a plurality of groups of training data, wherein each group of training data in the plurality of groups of training data comprises an image and a marked edge line of the image, and the image is shot in different scenes; training an initial network model by utilizing a plurality of groups of training data, and determining a predicted edge line; determining a loss value of the initial network model based on the predicted edge line and the marked edge line; in response to the loss value being greater than a predetermined threshold, updating a network parameter of the initial network model based on the loss value; determining the initial network model as the target network model in response to the loss value being less than or equal to the predetermined threshold. In this embodiment, when determining the edge line, the determination may also be performed by using the target network model. The target network model may be trained using multiple sets of training data. Each set of training data includes an image and a labeled edge line of the image. The target network model may be SSD, YOLO, Fast-rcnn detection network model, etc., which is not limited by the present invention.
In the above embodiment, a YOLO detection network model is taken as an example to explain: in the model training stage, image data with edge line labels can be input into a detection network for training, the image data is processed by the network to generate a characteristic matrix, the matrix is mapped into a target position and category prediction result in an original image through the YOLO standard post-processing, the result and the original label result are subjected to loss calculation and back propagation, and loss is reduced through continuous iteration, so that the training and optimization of network model parameters are realized. After the model training is finished, the model can be deployed to a network camera, the camera receives a frame of image signal, then the image is input to a detection network, the image is processed by the detection network to generate a characteristic matrix, and a prediction result of the position of the edge line in the original image is generated through YOLO standard post-processing, so that the automatic detection of the road is realized, and the edge line is determined.
In the above-described embodiment, the images included in the training data may be images taken in different scenes, and thus, the target network model may identify edge lines in different scenes. The method and the device realize the self-adaptive detection of the images under different scenes, automatically generate the regular guard lines or guard zones under various scenes, and the guard lines or the guard zones under different scenes can be different, so that the abnormal behaviors and the illegal behaviors can be different.
In the above embodiment, when the target scene of the target image is a breeding scene and the target object is an animal, referring to fig. 4, the behavior diagram of the target object is determined, as shown in fig. 4, an upper edge of a fence in a video may be automatically detected by the target network model, and a fence warning line (as shown by a thick solid line in fig. 4) may be automatically generated according to the detection result; meanwhile, target objects in the video, such as common livestock animals like cattle, sheep, horses, etc., are detected, and as shown by a rectangular frame in fig. 4, the target objects are tracked and a motion track is generated. If the movement track of the target animal (corresponding to the target moving track) crosses the fence guard line and the distance from the guard line at a certain moment after crossing exceeds a preset distance d, the target object is considered to generate a fence crossing behavior (namely, a behavior of crossing the guard area), and the system generates an alarm of the animal fence crossing behavior. If the movement track of the target object crosses the fence guard line, but the distance between the target object and the guard line does not exceed the preset distance d after crossing, the target is considered to continuously wander in the fence boundary (corresponding to the border crossing behavior), and the system generates a warning that the animal wanders in the fence boundary; if the motion trail of the target object does not cross the fence warning line all the time, the target is considered not to generate the fence crossing behavior (corresponding to the non-border crossing behavior), and the system does not give an alarm. The system can automatically monitor the target object in real time and generate an alarm when the target animal crosses the fence, so that the pasture management efficiency is effectively improved.
In the above embodiment, the target scene of the target image may also be a parking prohibition scene, and when the target object is a vehicle, referring to fig. 5, as shown in fig. 5, the road surface prohibition sign in the video may be automatically detected by a deep learning detection model (i.e., a target network model), and a prohibition warning region is automatically generated according to the edge thereof (as shown in block 1 in fig. 5), and simultaneously, the detection frame of the vehicle in the video is detected, tracked, and the motion trajectory is generated. If the target vehicle track passes from the outside of the warning area to the inside of the warning area and the stay duration time in the warning area exceeds the threshold value t, the target vehicle is considered to generate illegal parking behaviors in the forbidden area, and the system carries out vehicle illegal parking alarm; if the target track passes from the outside of the warning area to the inside of the warning area and the track passes through the outside of the warning area again within the time less than the threshold value t, the target vehicle is considered to generate a passing behavior in the no-parking area, and the system does not give an alarm; if the target vehicle track does not generate any intersection with the warning area, the target vehicle is not considered to pass through the no-parking area, and the system does not give an alarm. The system can monitor and analyze whether vehicles illegally park in the no-parking area in real time and automatically, and gives an alarm when the vehicles illegally park, so that the management efficiency of the road system is improved.
In an exemplary embodiment, the images included in each of the training data sets further include an object and an object marker detection box, and determining the target movement trajectory of the target object included in the target image based on the target image includes: determining a current detection frame of the target object included in the target image based on the target network model, wherein the current detection frame is used for framing the target object; determining an intersection ratio of the current detection frame and each detection frame included in a historical detection frame, wherein the historical detection frame is a detection frame which is adjacent to the target image and corresponds to each object included in the image acquired before the target image is acquired; determining the target movement trajectory of the target object based on the intersection ratio. In this embodiment, the detection frame of the target object may also be obtained through a target network model. I.e. the network model determining the current detection box of the target object and determining the edge line may be the same network model. In the training process, the image data with the edge line label and the object label can be input into the initial network model for training, and the target network model capable of simultaneously detecting the target object detection frame and the edge line is obtained.
In the above embodiment, when the target object is a pedestrian, the pedestrian detection and the road detection share the same detection model, so as to shorten the detection time of the whole system. The training and prediction process of the model is similar to the detection of the determined edge line, and the output category is pedestrians. The white rectangular frame in fig. 3 is the current detection frame of the pedestrian target detected by the system.
In one exemplary embodiment, determining the target movement trajectory of the target object based on the intersection ratio includes: determining a maximum cross-over ratio included in the cross-over ratio; in response to the fact that the maximum intersection ratio is larger than a preset intersection ratio, determining a historical movement track corresponding to a target historical detection frame corresponding to the maximum intersection ratio, and updating the current detection frame into the historical movement track to obtain the target movement track; and in response to the fact that the maximum intersection ratio is smaller than or equal to the preset intersection ratio, creating a moving track, and updating the current detection frame into the created moving track to obtain the target moving track. In this embodiment, when the target movement track is determined, a maximum intersection ratio included in the intersection ratio may be determined, and when the maximum intersection ratio is greater than a preset intersection ratio, a history movement track corresponding to a target history detection frame corresponding to the maximum intersection ratio is determined, and the current detection frame is updated to the history movement track, so as to obtain the target movement track.
In the above embodiment, based on the detection frame determined by the pedestrian detection result, a multi-target tracking method may be adopted to perform frame ID matching before and after each detected pedestrian and generate a motion trajectory. Specifically, the IOU values between the current frame target object detection frame A and all the previous frame history detection frames are calculated, namely, the intersection ratio is calculated, and the set { I is formed1,I2,…,IkK is the number of the target objects detected in the previous frame. Let the maximum value in the set be IjCorresponding to the detection frame J in the previous frame of image, then when I isjAnd when the distance is larger than or equal to the threshold t (corresponding to the preset intersection ratio), the matching between the A and the J is considered to be successful, and the central point of the A is taken as the latest track end point and is updated into the motion track corresponding to the J, so that the real-time tracking of the target motion track is realized.
In the above embodiment, if Ij is smaller than the preset intersection ratio t, the matching is considered to be failed. The preset intersection represents the strictness degree of successful matching of the front and rear frame detection frames compared with t, the larger t is, the stricter the matching requirement is, and the smaller t is, the looser the matching requirement is. The preset intersection ratio t can be manually adjusted according to requirements in the using process. In the pedestrian tracking stage, the motion trail corresponding to each detection frame has 4 states: create, update, lost, delete. If the detection frame appears in the first frame of the video, or appears in the second frame and later and is failed to be matched with all the detection frames of the previous frame, the detection frame is considered as a newly-appeared target, and the corresponding track state is set as create; if the detection frame is successfully matched with a certain detection frame of the previous frame, updating the central point of the detection frame serving as a new track end point into the successfully matched motion track, and setting the track state as update; if the motion track is not matched with any detection frame in the current frame, the tracked target is considered to be in a lost state, and the track state is set to be lost; for a track in the lost state, if matching succeeds at least once in the next continuous frames, the state of the track is reset to update, otherwise, the track is considered to not belong to any target any more, the state of the track is set to delete, and the track is deleted.
In an exemplary embodiment, after determining the behavior of the target object based on the position relationship of the guard line and the movement trajectory, the method further includes: performing an alert operation in response to a presence of at least one of the following behavior of the target object: and passing through alert zone behaviors, border crossing behaviors, alert zone crossing behaviors and illegal stay behaviors. In this embodiment, when the target object is a pedestrian and the alert area is a road, if the moving track of the pedestrian crosses the alert line and the distance between the moving track of the pedestrian and the alert line at a certain moment after crossing exceeds a predetermined distance, the pedestrian is considered to have a behavior of crossing the road, the system generates a warning that the pedestrian crosses the road, and reports the position of the pedestrian in the form of a detection frame; if the moving track of the pedestrian crosses the warning line, but the distance between the moving track of the pedestrian and the warning line does not exceed the preset distance d all the time after crossing, the pedestrian is considered to pass to the inner side of the road from the sidewalk and is continuously positioned at the inner side of the road, the system generates pedestrian boundary crossing alarm, and the position of the pedestrian is reported in a detection frame mode; and if the pedestrian movement track does not cross the warning line, the pedestrian is considered not to cross the road, and the alarm processing is not carried out. If the pedestrian movement track passes through 2 warning lines in sequence, the pedestrian is considered to completely cross the road, the system generates an alarm that the pedestrian completely crosses the road, and the position of the pedestrian is reported in a detection frame mode; if the motion track of the pedestrian crosses one of the warning lines and the distance between the pedestrian and the warning line at a certain moment after crossing exceeds a preset distance d, the pedestrian is considered to have a road crossing behavior, the system generates a road crossing warning for the pedestrian and reports the position of the pedestrian in a detection frame mode; if the moving track of the pedestrian crosses one of the warning lines, but the distance between the pedestrian and the warning line does not exceed the preset threshold distance d all the time after crossing, the pedestrian is considered to pass to the inner side of the road from the sidewalk and is continuously positioned at the inner side of the road, the system generates pedestrian out-of-range alarm, and the position of the pedestrian is reported in a detection frame mode; and if the pedestrian movement track does not cross any warning line, the pedestrian is considered not to cross the road, and the alarm processing is not carried out.
In the above embodiments, the alerting operation may include using an audible alert, for example, an alarm may be played when the target object is behaving as one or more of passing alert zone behaviour, border crossing behaviour, crossing alert zone behaviour, and violation of a parking action. The alerting operation may also include a light alert, for example, where the target object is behaving as passing one or more of an alert zone behaviour, an out-of-range behaviour, a crossing alert zone behaviour, an offending stop behaviour, the projection device may be controlled to project light onto the alert line to alert the target object to return to a safe location.
The following describes a method for determining the behavior of an object with reference to a specific embodiment, in this embodiment, the target object is a pedestrian, and the alert area is a road:
fig. 6 is a flowchart of a method for determining object behavior according to an embodiment of the present invention, and as shown in fig. 6, the flowchart includes:
step S602, inputting a frame of image;
step S604, road detection;
step S605, generating a warning line;
step S606, pedestrian detection;
step S608, pedestrian tracking;
step S610 of determining whether or not a warning line is present, and if yes, performing step S612, and if no, performing step S620;
step S612, judging whether the pedestrian track crosses the warning line, if so, executing step S614, and if not, executing step S602;
in step S614, it is determined whether the distance between the end point of the trajectory and the guard line is greater than the distance d, and if the determination result is yes, step S616 is executed, and if the determination result is no, step S618 is executed.
Step S616, generating pedestrian crossing road behavior alarm;
and step 618, generating a pedestrian boundary crossing behavior alarm.
Step S620, judging whether the pedestrian track crosses two warning lines, if not, executing step S622, and if yes, executing step S630;
step S622, judging whether the pedestrian track crosses a warning line, if so, executing step S624, and if not, executing step S602;
step S624, judging whether the distance between the track end point and the warning line is greater than d, if yes, executing step S626, and if no, executing step S628;
step S626, generating pedestrian crossing road behavior alarm;
step S628, generating pedestrian border crossing behavior alarm;
and step S630, generating a pedestrian complete crossing road behavior alarm.
In the embodiment, based on the deep learning technology, the deep semantic features of the target are extracted to detect and track the pedestrians and the road, and then whether the pedestrians cross the road is detected through the position relation between the pedestrian track and the road warning line, so that the behavior detection of the pedestrians crossing the road is realized in various complex environments, and the robustness and the generalization performance are better. The automatic generation of the warning lines is realized through the road edge detection, the warning lines can be automatically generated under various different scenes, and the labor cost is reduced. The method is not limited to the road warning line, but also can be applied to warning line or warning area generation of target areas such as an enclosure, a fence, a zebra crossing, a no-parking area and the like, and different logic designs and judgments can be carried out based on actual requirements, so that more behavior types can be intelligently analyzed.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
In this embodiment, a device for determining an object behavior is further provided, and the device is used to implement the foregoing embodiments and preferred embodiments, and details of which have been already described are omitted. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 7 is a block diagram of a structure of an apparatus for determining a behavior of an object according to an embodiment of the present invention, as shown in fig. 7, the apparatus including:
an obtaining module 72, configured to obtain a target image obtained by shooting a target area;
a first determining module 74 for determining a warning line included in the target area based on the target image;
a second determining module 76, configured to determine a target moving trajectory of a target object included in the target image based on the image sequence in which the target image is located;
a third determining module 78, configured to determine the behavior of the target object based on the position relationship between the warning line and the target moving trajectory.
In an exemplary embodiment, the third determining module 78 may determine the first distance between the end point of the target movement trajectory and the warning line by; determining a behavior of the target object based on the first distance.
In an exemplary embodiment, the third determining module 78 may implement the determining the first distance between the end point of the target movement track and the warning line by: in response to the position relationship indicating that the target movement track and the warning line have an intersection, determining whether an end point of the target movement track is updated within a first predetermined time based on an image captured after the target image is captured; in response to the endpoint not being updated, determining a first distance of the endpoint from the warning line.
In an exemplary embodiment, the alert line includes a no-pass alert line, and the third determination module 78 may implement the determining the behavior of the target object based on the first distance by: in response to the first distance being greater than a predetermined distance, determining the behavior of the target object as being a passing alert zone behavior; or in response to the first distance being less than or equal to the predetermined distance, determining the behavior of the target object as out-of-range behavior.
In an exemplary embodiment, the alert line includes a stay-prohibited alert line, and the third determination module 78 may implement the determining the behavior of the target object based on the first distance by: in response to determining that the target object enters an alert zone determined based on the alert line based on the first distance, determining a time at which the target object enters the alert zone; and in response to the time being greater than a second predetermined time, determining that the behavior of the target object is an illegal stay behavior.
In an exemplary embodiment, the alert line includes a plurality of no-pass alert lines, and the third determination module 78 may determine the behavior of the target object based on the position relationship of the alert line to the target movement trajectory by: responding to the position relation to indicate that the target moving track and a plurality of warning lines included in the warning lines have intersection points; determining the behavior of the target object as a behavior crossing an alert zone.
In an exemplary embodiment, the warning line is determined based on an edge line of the target zone.
In one exemplary embodiment, the apparatus may be configured to detect a target scene included in the target image and generate an edge line of the target area based on the target scene before determining a warning line included in the target area based on the target image; determining the warning line based on the margin line.
In an exemplary embodiment, the apparatus may detect a target scene included in the target image and generate an edge line of the target area based on the target scene by: determining the target scene included in the target image based on the trained target network model, and determining the edge line based on the target scene, wherein the target network model is trained in the following way: acquiring a plurality of groups of training data, wherein each group of training data in the plurality of groups of training data comprises an image and a marked edge line of the image, and the image is shot in different scenes; training an initial network model by utilizing a plurality of groups of training data, and determining a predicted edge line; determining a loss value of the initial network model based on the predicted edge line and the marked edge line; in response to the loss value being greater than a predetermined threshold, updating a network parameter of the initial network model based on the loss value; determining the initial network model as the target network model in response to the loss value being less than or equal to the predetermined threshold.
In an exemplary embodiment, the images included in each training data set included in the plurality of training data sets further include an object and an object mark detection box, and the second determining module 76 may determine the target moving track of the target object included in the target image based on the target image by: determining a current detection frame of the target object included in the target image based on the target network model, wherein the current detection frame is used for framing the target object; determining an intersection ratio of the current detection frame and each detection frame included in a historical detection frame, wherein the historical detection frame is a detection frame which is adjacent to the target image and corresponds to each object included in the image acquired before the target image is acquired; determining the target movement trajectory of the target object based on the intersection ratio.
In an exemplary embodiment, the second determining module 76 may determine the target movement trajectory of the target object based on the intersection ratio by: determining a maximum cross-over ratio included in the cross-over ratio; in response to the fact that the maximum intersection ratio is larger than a preset intersection ratio, determining a historical movement track corresponding to a target historical detection frame corresponding to the maximum intersection ratio, and updating the current detection frame into the historical movement track to obtain the target movement track; in response to the maximum intersection ratio being less than or equal to the preset intersection ratio, creating a movement track; and updating the current detection frame to the created movement track to obtain the target movement track.
In an exemplary embodiment, the apparatus may be further configured to, after determining the behavior of the target object based on the position relationship of the warning line and the movement trajectory: performing an alert operation in response to a presence of at least one of the following behavior of the target object: is passing alert zone behavior, border crossing behavior, crossing alert zone behavior.
It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.
Embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method as set forth in any of the above.
In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
In an exemplary embodiment, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary embodiments, and details of this embodiment are not repeated herein.
It will be apparent to those skilled in the art that the various modules or steps of the invention described above may be implemented using a general purpose computing device, they may be centralized on a single computing device or distributed across a network of computing devices, and they may be implemented using program code executable by the computing devices, such that they may be stored in a memory device and executed by the computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into various integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.
Claims (15)
1. A method for determining behavior of an object, comprising:
acquiring a target image obtained by shooting a target area;
determining a warning line included in the target area based on the target image;
determining a target moving track of a target object included in the target image based on the image sequence in which the target image is located;
and determining the behavior of the target object based on the position relation between the warning line and the target moving track.
2. The method according to claim 1, wherein the determining the behavior of the target object based on the position relationship of the warning line and the target movement track comprises:
determining a first distance between the end point of the target moving track and the warning line;
determining a behavior of the target object based on the first distance.
3. The method of claim 2, wherein said determining a first distance between an end point of said target movement trajectory and said warning line comprises:
in response to the position relationship indicating that the target movement track and the warning line have an intersection, determining whether an end point of the target movement track is updated within a first predetermined time based on an image captured after the target image is captured;
in response to the endpoint not being updated, determining a first distance of the endpoint from the warning line.
4. The method according to any of claims 2 to 3, wherein the watch line comprises a no-pass watch line, and wherein the determining the behavior of the target object based on the first distance comprises:
in response to the first distance being greater than a predetermined distance, determining the behavior of the target object as being a passing alert zone behavior; or
Determining that the behavior of the target object is out-of-range behavior in response to the first distance being less than or equal to the predetermined distance.
5. The method according to any of claims 2 to 3, wherein the alert line comprises a stay-prohibited alert line, and wherein the determining the behavior of the target object based on the first distance comprises:
in response to determining that the target object enters an alert zone determined based on the alert line based on the first distance, determining a time at which the target object enters the alert zone;
and in response to the time being greater than a second predetermined time, determining that the behavior of the target object is an illegal stay behavior.
6. The method according to claim 1, wherein the alert line comprises a plurality of no-go alert lines, and wherein determining the behavior of the target object based on the positional relationship of the alert line to the target movement trajectory comprises:
responding to the position relation to indicate that the target moving track and a plurality of warning lines included in the warning lines have intersection points;
determining the behavior of the target object as a behavior crossing an alert zone.
7. The method according to any one of claims 1 to 6, wherein the alert line is determined based on an edge line of the target zone.
8. The method according to claim 1, wherein prior to determining a warning line included in the target zone based on the target image, the method further comprises:
detecting a target scene included in the target image, and generating an edge line of the target area based on the target scene;
determining the warning line based on the margin line.
9. The method of claim 8, wherein detecting a target scene included in the target image and generating edge lines of the target region based on the target scene comprises:
determining the target scene included in the target image based on the trained target network model, and determining the edge line based on the target scene, wherein the target network model is trained in the following way:
acquiring a plurality of groups of training data, wherein each group of training data in the plurality of groups of training data comprises an image and a marked edge line of the image, and the image is shot in different scenes;
training an initial network model by utilizing a plurality of groups of training data, and determining a predicted edge line;
determining a loss value of the initial network model based on the predicted edge line and the marked edge line;
in response to the loss value being greater than a predetermined threshold, updating a network parameter of the initial network model based on the loss value;
determining the initial network model as the target network model in response to the loss value being less than or equal to the predetermined threshold.
10. The method of claim 9, wherein the images included in each set of training data included in the plurality of sets of training data further include objects and an object marker detection box, and wherein determining the target movement trajectory of the target object included in the target image based on the target image includes:
determining a current detection frame of the target object included in the target image based on the target network model, wherein the current detection frame is used for framing the target object;
determining an intersection ratio of the current detection frame and each detection frame included in a historical detection frame, wherein the historical detection frame is a detection frame which is adjacent to the target image and corresponds to each object included in the image acquired before the target image is acquired;
determining the target movement trajectory of the target object based on the intersection ratio.
11. The method of claim 10, wherein determining the target movement trajectory of the target object based on the intersection ratio comprises:
determining a maximum cross-over ratio included in the cross-over ratio;
in response to the fact that the maximum intersection ratio is larger than a preset intersection ratio, determining a historical movement track corresponding to a target historical detection frame corresponding to the maximum intersection ratio, and updating the current detection frame into the historical movement track to obtain the target movement track;
and in response to the fact that the maximum intersection ratio is smaller than or equal to the preset intersection ratio, creating a moving track, and updating the current detection frame into the created moving track to obtain the target moving track.
12. The method according to claim 1, wherein after determining the behavior of the target object based on the positional relationship of the guard line and the movement trajectory, the method further comprises:
performing an alert operation in response to a presence of at least one of the following behavior of the target object: is passing alert zone behavior, border crossing behavior, crossing alert zone behavior.
13. An apparatus for determining behavior of an object, comprising:
the acquisition module is used for acquiring a target image obtained by shooting a target area;
a first determination module for determining a warning line included in the target area based on the target image;
a second determining module, configured to determine a target movement trajectory of a target object included in the target image based on an image sequence in which the target image is located;
and the third determination module is used for determining the behavior of the target object based on the position relation between the warning line and the target moving track.
14. A computer-readable storage medium, in which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the method of one of claims 1 to 12.
15. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111545195.4A CN114255424A (en) | 2021-12-16 | 2021-12-16 | Method and device for determining object behaviors, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111545195.4A CN114255424A (en) | 2021-12-16 | 2021-12-16 | Method and device for determining object behaviors, storage medium and electronic device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114255424A true CN114255424A (en) | 2022-03-29 |
Family
ID=80792660
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111545195.4A Pending CN114255424A (en) | 2021-12-16 | 2021-12-16 | Method and device for determining object behaviors, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114255424A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114973573A (en) * | 2022-06-14 | 2022-08-30 | 浙江大华技术股份有限公司 | Target intrusion determination method and device, storage medium and electronic device |
| CN116092023A (en) * | 2023-02-03 | 2023-05-09 | 以萨技术股份有限公司 | Data processing system for determining abnormal behaviors |
-
2021
- 2021-12-16 CN CN202111545195.4A patent/CN114255424A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114973573A (en) * | 2022-06-14 | 2022-08-30 | 浙江大华技术股份有限公司 | Target intrusion determination method and device, storage medium and electronic device |
| CN116092023A (en) * | 2023-02-03 | 2023-05-09 | 以萨技术股份有限公司 | Data processing system for determining abnormal behaviors |
| CN116092023B (en) * | 2023-02-03 | 2023-10-20 | 以萨技术股份有限公司 | Data processing system for determining abnormal behaviors |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109686088B (en) | Traffic video alarm method, equipment and system | |
| CN114255424A (en) | Method and device for determining object behaviors, storage medium and electronic device | |
| CN109377694B (en) | Monitoring method and system for community vehicles | |
| Chang et al. | Video analytics in smart transportation for the AIC'18 challenge | |
| CN111753612B (en) | Method and device for detecting casting object and storage medium | |
| CN114973573A (en) | Target intrusion determination method and device, storage medium and electronic device | |
| US20220237919A1 (en) | Method, Apparatus, and Computing Device for Lane Recognition | |
| CN111126323A (en) | Bayonet element recognition and analysis method and system serving for traffic violation detection | |
| Zheng et al. | YOLO-BYTE: An efficient multi-object tracking algorithm for automatic monitoring of dairy cows | |
| CN113269042B (en) | Intelligent traffic management method and system based on driving vehicle violation identification | |
| CN112733598A (en) | Vehicle law violation determination method and device, computer equipment and storage medium | |
| CN111929672A (en) | Method and device for determining movement track, storage medium and electronic device | |
| CN112447060A (en) | Method and device for recognizing lane and computing equipment | |
| WO2024098992A1 (en) | Vehicle reversing detection method and apparatus | |
| CN111814668B (en) | Method and device for detecting road sprinklers | |
| Kahlon et al. | An intelligent framework to detect and generate alert while cattle lying on road in dangerous states using surveillance videos | |
| CN111047864A (en) | Warning method, warning equipment and computer storage medium for running red light | |
| CN108922175B (en) | Method and device for recording illegal behaviors of multiple motor vehicles crossing solid line | |
| CN116311166A (en) | Traffic obstacle recognition method and device and electronic equipment | |
| CN111274336B (en) | Target track processing method and device, storage medium and electronic device | |
| CN112801200B (en) | Data packet screening method, device, equipment and storage medium | |
| CN115050047A (en) | Method and device for protecting animals in transformer substation, electronic equipment and storage medium | |
| CN112580482A (en) | Animal monitoring method, terminal and storage medium | |
| CN113221800A (en) | Monitoring and judging method and system for target to be detected | |
| CN116863298B (en) | Training and early warning sending method, system, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |