KR20180072547A - Method for and apparatus for detecting events - Google Patents

Abstract

The present invention relates to a method for detecting events with less processing power. The method comprises the steps of: repeatedly registering values indicating the amount of data generated by an encoder encoding a video from a scene by time compression; determining if a particular event has occurred in the scene represented by the encoded video by comparing properties of the registered values with predetermined properties; and generating an event signal in response to an event occurrence being determined.

Description

[0001] METHOD AND APPARATUS FOR DETECTING EVENTS [0002]

The present invention relates to event detection from captured video. More particularly, the present invention relates to a method and apparatus for detecting events in captured video.

Most of today's monitoring, monitoring and security systems include devices that capture and generate video. The captured video is often analyzed to detect the occurrence of certain events. Examples of such events are motion detection, object detection, and trespass detection. Detecting these types of events requires a lot of processing power. These high throughput requirements can limit the possibility of providing event detection in low throughput cameras that capture video. One reason for using low throughput cameras may be due to the high price of high throughput cameras. Another reason is that a camera with high throughput requires more power than a camera with low throughput and may not be able to provide power to the location of the camera. Thus, there is a need for a method of generating event detection requiring less computing power.

One object of the present invention is to detect events with less processing power.

This object is achieved by a method for detecting events according to claim 1. Other embodiments of the invention are set out in the dependent claims.

Specifically, in accordance with some embodiments of the present invention, a method of detecting events includes repeatedly registering a value representing an amount of data generated by an encoder that encodes video from a scene by temporal compression, Comparing the characteristics of the values with predetermined characteristics to determine if a particular event has occurred in the scene represented by the encoded video, and generating an event signal in response to the event occurrence being determined. By using this data to register the amount of data generated in the encoding of the video and then to determine if an event has occurred in the captured video, the determination can be performed without performing a strong analysis of all the image data Processing power can be saved. There may also be more advantages in the fact that there is less data needed for the decision. That is, the transmission of data for analysis may require less storage capacity to reduce network load, or to store data that can determine whether an event has occurred or a particular event has occurred.

In other embodiments, the characteristics to be compared in the act of determining whether a particular event has occurred are that the registered values are increased at a rate higher than the first predetermined threshold, and the at least one registered value is a second predetermined threshold Value, the registered values are reduced at a rate greater than the third predetermined threshold, and the integration of values over a period of time exceeds a fourth predetermined threshold value, or May be any combination of characteristics. One advantage of these properties is that it does not require much processing power or memory to be computed from registered values that represent the amount of data generated by the encoder.

Moreover, a particular event can be determined and identified by matching a characteristic representing a sequence of registered values with a predetermined event characteristic based on a description indicating a sequence of corresponding values. An advantage of the matching sequences of registered values may be that more specific events and / or more complex events can be detected and identified.

In yet another embodiment, an event is determined by comparing characteristics representing a sequence of registered values with a description representing a steady state, and such event may be determined by determining whether the sequence of registered values is greater than a predetermined value It is judged that it occurs when it is different from the steady state. An advantage of determining if an event has occurred if the characteristics differ from the steady state characteristics is that the determination of a number of events, such as an unexpected event or an unknown event, can be detected relative to several characteristics, .

In accordance with other embodiments, each repeatedly registered value representing the amount of data generated by the encoder is determined by measuring the amount of transmitted data comprising data generated by the encoder from the time of the previously registered value . Moreover, the measurement of the transmitted data can be performed from the time of the last registered value.

Some embodiments may further comprise storing the registered values in a database and the operation of determining whether a particular event has occurred has been performed on values registered in the database.

Still other embodiments may further comprise transmitting the registered values to the client over the network and the operation of determining whether a particular event has occurred has been performed by the client.

In some embodiments, the property of the registered values indicating an increase in value over time indicates an object entering the scene in the encoded video. The advantage of this feature is that it can determine the characteristics of more complex events.

In other embodiments, the property of the registered values is a value itself, and a first object type moving through the scene is identified by comparing the registered value with a predetermined first threshold value, The second type of object moving through is identified by comparing the registered value with a predetermined second threshold value.

According to some embodiments, the operation of determining whether a particular event has occurred is performed only for registered values that are greater than a predetermined threshold, and the benefit of this determination method is very simple and requires very low processing power to operate It is.

In some embodiments, values representing data comprising image frames that are not temporally encoded are not used in the operation of determining whether a particular event has occurred. An advantage of not determining events based on non-temporally encoded image frames is that the non-temporally encoded frames do not carry any information with respect to motion in the video, More processing power is saved.

In further embodiments, the method further comprises incrementing the counter each time an occurrence of a particular event is determined. The advantage of this is that counters that require low throughput, e.g., counters for people, counters for vehicles, and generic objects, can be implemented. Such a counter may be implemented, for example, in a camera with limited resources or limited resources, or at least in another device with limited spare resources. In some embodiments, the method further comprises incrementing a counter associated with a particular event each time an occurrence of a particular event is determined. Thus, a counter for specific events is placed, which means that different counters can be used for different objects such as, for example, people, animals, vehicles, etc., or different events such as running, For example.

Further scope of applicability of the present invention will become apparent from the following detailed description. It should be understood, however, that the detailed description and specific examples, while indicating the preferred embodiments of the invention, are given by way of illustration only, since various changes and modifications within the scope of the invention will be apparent to those skilled in the art from this detailed description. Accordingly, it is to be understood that the invention is not limited to the specific component parts of the described device or the described method steps, as such devices and methods may vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting. As used in the specification and appended claims, the terms " a "and" above "are intended to indicate that there is one or more of the elements present in the context unless otherwise indicated explicitly in the context. Thus, for example, reference to "sensor" or "sensor" may include various sensors and the like. Also, the word "comprising " does not exclude other elements or steps.

Other features and advantages of the present invention will become apparent from the following detailed description of presently preferred embodiments with reference to the accompanying drawings.
1 is a schematic diagram of a system that may include an embodiment of the present invention.
2 is a block diagram of a camera according to an embodiment of the present invention.
3 is a graph illustrating data rates over time showing features of a graph that may be used in an embodiment of the present invention.
4 is a flow diagram of a process for generating event signals in accordance with an embodiment of the present invention.
Also, in the drawings, like reference numerals refer to like or corresponding parts throughout the several views.

The present invention relates to an event detection system and an event detection method. The event detection system may be a system including a plurality of monitoring cameras 12a-12c, a communication network 14, systems management devices 16a-16b, and a data storage device 18 (see Figure 1) . The monitoring cameras 12a-12c may be any type of network monitoring camera capable of capturing motion video. The communication network 14 may be any wired or wireless communication network or it may be a combination of various wired and / or wireless networks, e.g., LAN, WAN, Internet, cellular network, Wi-Fi, The system management devices 16a-b may be any computing device having a display that allows an operator to interact with a device in the system or receive information related to event detection. The data storage device 18 may be a network attached storage unit (NAS), a file server, a video server, or any other data storage unit.

Referring now to FIG. 2, the monitoring camera 12 used in the system is configured to capture the motion video of the scene as digital data. Thus, the monitoring camera 12 includes at least one lens 40 that directs light from the scene onto the image sensor 42. The image sensor 42 and the image processor 44 convert the light into digital data representing the captured image. The camera 12 is also configured to capture a plurality of images to provide data for generating motion video. The data representing the motion video is then encoded in an encoder 46 that encodes the motion video using a temporal compression scheme. The camera may be a camera that captures visible light, an IR camera, a ToF camera (flight time camera), and so on.

The time compression scheme reduces the size of the compressed video by not encoding each frame into a separate complete image. Fully encoded frames are referred to as intra frames because they are fully encoded based on information of the currently encoded image frame. All other frames in the video are represented by data specifying changes after the last frame. That is, the data representing these frames represents differences from other frames. These frames are referred to as delta interframes because they depend on the data of the other frames. Inter frames may be associated with previous frames referred to as P frames in some encoding schemes, but alternatively may be associated with previous and future frames referred to as B frames.

The encoding of the motion video in the encoder 46 reduces the amount of data representing the motion video. Examples of encoding schemes used in the encoder 46 are H.264, H.265, MPEG-2 Part 2, MPEG-4 Part 2, and the like. In some cameras 12, the motion video is much more analyzed and / or processed before and / or after the encoding of the motion video. In these cameras 12, the processing unit 48 may be deployed for this type of analysis and / or processing. Many cameras 12 also require volatile memory 50 and / or non-volatile memory 52 for the operation of programs and storage of data.

Before the motion video is transmitted over the network 14, the video data is managed by a communication protocol and then physically adapted to the network connected to the camera. The network interface 54 of FIG. 2 may be viewed as a device or module that performs these operations in any manner known to those of ordinary skill in the art. The communication between devices and modules within the camera may be implemented by direct individual physical connections or a data bus or a combination thereof and these communication implementations are symbolized in FIG. 2 by an internal communication path 56.

The event detection system of the present invention is designed to detect events using less processing power than systems that analyze all pixels of a video stream to detect events. When detecting events of stored data, i. E., Indicating old recordings, event detection of data representing a longer period of time will be faster than when using a system that analyzes all pixels of a video stream to detect events. These advantages are achieved by analyzing the amount of data generated by the temporally compressing video encoder 46 of the camera 12 capturing the motion video of the scene for which event detection is desired. The amount of data generated by the encoder 46 depends on the amount of change between successively captured image frames. For example, an image sequence of a scene that does not have essentially moving objects will result in less encoded data than an image sequence of a scene with many motions. Thus, the amount of data after the encoding step largely depends on the variants of successively captured images. This may also be described as the amount of encoded image data generated by the encoder 46 depending on the number of pixels on the image sensor that are affected by the movements in the scene. For example, an object close to a camera or a large object affects more pixels on the image sensor than objects far from the camera or a small object. Thus, the amount of data is the effect of temporal compression of the image information.

According to embodiments of the present invention, a plurality of data rates representing the amount of data per unit time in the data stream including the motion video of the monitored scene are continuously measured and registered. As a result, a data set indicating a change in data rate over time is registered. The data rate may be measured or retrieved at various locations in the system. For example, the data rate may vary depending on the camera encoder 46, the network interface 54 of the camera, the switch or router of the network 14, the network interface or decoder of the system management device 16a-16b, Lt; RTI ID = 0.0 > 18 < / RTI > The data rate can be measured directly on the stream representing the image stream, but can also be measured by measuring the data rate of the network packets transmitting the motion video. The process of measuring the data rate of a data stream or any data transmission is well known to those of ordinary skill in the art. The data set representing the data rate over time can be expressed using a very small amount of data, in particular, compared to the data needed to represent the motion video stream. For example, consideration should be given to including a value or data entry indicating the amount of data used to represent each image of the image stream in a data set that represents a change in data rate over time. If the camera is a 10 megapixel camera, that is, the size of the spatially encoded intra frame will be about 2.5 MB using any of the standard encoders. Even 10 megapixel frames encoded using lossless coding schemes do not have a size much larger than 10 MB. To register a value or data item that represents any size between 0 MB and 10 MB, only three bytes are required, which can represent more than 16 million values if only three bytes are present. If the resolution of the data rate is not so important, 2 or 1 byte may be used to indicate the data rate. In any case, the data required to represent the data rate in the data set is about one million times smaller than the data used to represent one intra frame. Since the interframe is temporally encoded in streaming video, the data used per frame is of course smaller. However, the data set used need not include the size of every frame, but may represent the amount of accumulated data over a predetermined period of time. In one embodiment, the frame size is registered as a representation of the data rate. The frame size represents the amount of data needed to encode the frame. The frame size is registered using 4 bytes (32 bits) and the frame size is registered for each frame. The frames are captured at 30 frames per second, and thus the data used in the present invention for the detection of events is 0.12 Kb / s in this embodiment. Which is much lower than the data rate of a standard motion video stream of 6000 Kb / s.

Apart from the image encoding of the image stream and the resolution, or number of bytes, used in the data set to represent each value or data item in the data set, when a data rate data set can be used instead of images in the image stream, The difference in the amount of data that must be spent is enormous. Because there is a need to store only a portion of the data, there is a huge storage advantage. For example, the monitoring system can store motion video data only for the period of time when interesting events occur and store the data rate value for the entire monitored period. The stored data rate values may be retrieved and / or analyzed to filter certain events. The data rate values can then also be used to confirm that no event has occurred during the period in which no motion video is stored. Also, there is generally the advantage that stored data rate data sets can be used to retrieve recorded video for video sequences of interest. Another advantage of analyzing a set of data rates is that events in a scene can be analyzed without compromising the integrity of the person in the scene.

In Figure 3, an example of a data set is shown by plotting the data rate over time. The data set of the drawing can be obtained by registering the amount of data representing each image frame in the video stream captured by the camera. The effect of intra frames in the current data set has been removed by simply removing each value representing the intra frame. The removal of intra-frames may alternatively be implemented by removing all data values that exceed the maximum threshold. An advantage of removing intra frames is that data rates can be analyzed more simply when there is no need to consider the peaks at the data rate of each intra-frame. In particular, since an intra frame is spatially encoded and is not temporally encoded, an intra frame does not contribute to information about movement or changes between frames. In this way, the data set will use a unique motion representation of the inter-frames in the time encoders without interference from the value of the high data amount of intra-frames.

The monitored scene shown in the graph of Fig. 3 is the road and the camera is arranged to capture the video above the road in the longitudinal direction of the road. You can identify various events by simply viewing the graph. For example, the height of each peak indicates the size of the vehicle when passing through the camera at the nearest point, at least the relative size of the vehicle. Thus, by simply focusing on the peak values of the graph, conclusions can be drawn about the size of the vehicles passing by the camera. Moreover, the steep slopes of the graphs on each side of the peak indicate the running direction of the vehicle. The steepest part of the graph is the result of a camera that either leaves the scene or encodes the video of a vehicle reaching the scene at a point close to the camera. That is, when the captured image of the vehicle is large, when leaving or arriving at the scene, the vehicle causes many pixels to change in a short time. The less steep portion of the graph on the other side of the peak is the result of a vehicle moving away from the captured scene or a vehicle driving away from the camera in the scene. It is perceived that as the vehicle has been in the scene for a relatively long time, it gradually becomes smaller as it moves away from the camera.

Now, referring to graph 104 of FIG. 3, from the above inference, the first peak 106 will represent a large car, such as a van or truck, driving towards the camera. This can be identified as representing a vehicle moving a long time in the scene until a portion of the graph 108 that follows up to the peak 106 reaches a location close to the camera, thereby inferring from the appearance of a less steep graph. The fact that the less steep portion of the graph 108 is registered before the top of the peak 106 of the graph indicates that the vehicle is moving in the scene towards the camera. Thus, the portion of the graph 110 on the subsequent side of the peak is a steep grade indicating that the vehicle leaves the scene close to the camera, as described above.

The second peak 112 represents a car that drives to move away from the camera in the scene. The size of the car is identified by the height of the peak 112 and the direction of travel is identified by having the steep portion 114 of the graph leading to the peak 112 and the less steep portion 116 of the graph after the peak. Peak 118 represents another large vehicle moving away from the camera, while peaks 120 and 122 represent vehicles moving away from the camera, respectively.

The final peak 124 of the graph 104 is higher than the peak representing the larger vehicles and when looking at the slope of the portions of the graphs 126 and 128 on each side of the peak 124, have. These indications can be concluded that the peaks represent two vehicles moving in opposite directions.

Thus, the features of the graph 104 that represent the data rate of the temporally encoded video data can be used to identify or at least indicate certain events. For example, a threshold may be set to indicate an event, such as a motion in a scene. When the data rate exceeds the threshold, the event is triggered. Events representing objects moving towards or away from the camera can be identified by analyzing the derivative of portions of the graphs on each side of the peak, i. E. By calculating the steepness of these portions of the graph as described above. Such a slope value, e.g., a slope, may also be used to indicate a speed event. For example, the velocity of an object may be computed based on the slope of the graph. The area under the graph can also be used for analysis of events. This area can be used to determine the speed of the vehicle, the size of the vehicle, and many others.

Another alternative is to create models representing various identifiable events in a graph representing data rates over time and then using these models to identify specific events. These models can be formed by identifying and registering a combination of specific characteristics of a portion of the graph representing a particular event or by registering a typical event if the registered graph does not deviate by more than a predetermined amount from the registered typical graph, As this event. The models can also be performed by using deep learning, for example, by training a neural network that identifies specific events by inputting the data rate of the temporally encoded video.

Another application is to create a model representing events that typically should occur in a scene and then trigger an alarm when the registered data rate is out of this model. For example, this system can be used to survey railroad tracks. A model representing trains passing over a track is formed by analyzing the characteristics of the graph or by training the system using a neural network approach. The system can then be set up to indicate when an event that does not fit the model is detected. Thus, the data rate graph of events such as animals or people on a track can be used to detect animals or people on a track because it is likely to be significantly different from the model representing the trains.

Data rate analysis for the detection of events can be used in many applications. As mentioned above, the data rate analysis for sensing these events can be used for traffic monitoring and monitoring of train tracks. Other areas of use are process monitoring in factories. For example, counting products on conveyor belts may include counting data rate sequences, i. E., Some or a plurality of characteristics of a data rate over time or one or more sections of such data rate sequences If it is different from the "normal" property or section, for example, it indicates an inconsistency with the intended process, an alarm is generated. As described above, the characteristics can be represented by specific data points, e.g., data rate values, slopes, regions, and sections can be displayed in the shape of a graph over that period. In addition, methods for detecting events can be used to monitor buildings or buildings to detect suspicious activity or dangerous activity. Methods of detecting these events may also be used to control building automation systems such as lighting, heating, ventilation, and the like.

The above use cases are provided using real-time data rates of cameras or cameras. However, the data rate data may advantageously be stored for later analysis. Data does not require much storage space and can be very useful when you want to find a point in time for a particular event to study the video of that event. In this particular case, you can use video, but data rate data is much more efficient in event searching because there is much less data to analyze. Therefore, it is a great advantage to store and use data rate information for later use.

Turning now to the flow diagram of FIG. 4, there is shown a process 200 of embodiments of the present invention. The process includes registering successive values each representing a data rate from a particular point in time (step 202). These values may be based on information provided by the camera's encoder, values obtained by measuring the amount of data output from the encoder, values provided by the network interface, e.g., network traffic. Each value and / or series of values represents a characteristic of the registered values as indicated in the examples above, and is compared with corresponding predetermined properties representing at least one event (step 204). If the comparison indicates that the value or series of values is indicative of an event (step 206), an event signal is generated (step 208), and thereafter, the process registers additional values indicative of the data rate of the data generated by the encoder The process returns to step 202. The event signal may be an internal signal or a signal transmitted to other systems or devices. For example, a signal may represent an event that represents the detection of an object, which may be sent internally to the counter to count the number of objects. Moreover, the event signal may be an event, e.g., a signal carrying data of a medium sized vehicle passing 60 km / h in the west direction. Also, the event signal may be an alarm signal. A typical descriptor will recognize variations in a number of event signals. If the comparison does not indicate any occurrence of an event (step 206), the process returns directly to step 202 to register additional values indicating the data rate of the data generated by the encoder. This process can be performed on the actual data or on the stored data, for example, because the data is generated and values are generated. For example, data is stored on some storage device and parsed to detect events of stored data.

The comparison at step 204 may include any one or any combination of the features discussed with respect to the data rate graph of FIG. For example, a value of the data rate compared to a threshold indicating whether there is sufficient change in the scene to trigger a motion event may be a compared property. The characteristic may be a derivative of the changes associated with a single data input or a plurality of successive data rate values in the graph, i. Such a derivative may be compared with a predetermined threshold value to indicate an event. Another property that can be compared is the integer number of data rate values, i.e., the area under the graph. Such an area can be simply measured between two time points at which the data rate exceeds the threshold or over a predetermined time frame. The comparison may comprise comparing a set of properties with a predetermined set of characteristics, e.g., any combination of the above-mentioned characteristics. Moreover, a set of properties may include a plurality of properties from different viewpoints. For example, an event indicating that the car is moving towards or away from the camera can be detected by detecting the first derivative, then detecting the peak value, and finally detecting the second derivative. The set of these property values may be compared to a predetermined set of properties.

Furthermore, the system can be implemented by arranging all the functions of the camera's system. That is, the camera includes means for performing a process according to embodiments of the present invention. For example, the camera 12 may retrieve data rate values from the encoder 46, from the network interface, or from any other data rate extraction point of the camera, and to detect live events, And stores the data rate values in the memory 50, 52 for comparison with the properties set in the program executed by the camera to detect events from the camera. Thereafter, the camera may transmit the event identifier in response to the event detection.

Alternatively, the system may be implemented in any device connected to the camera over a network and receiving streaming motion video. Such devices, e.g., system management devices 16a-16b, data storage device 18, etc., can measure the amount of data received in the motion video stream and execute the process on its own. The device may also be a storage device such as a file server, NAS, etc. configured to store the data rate retrieved from the data for storage or the data rate retrieved from the datagrams retrieved from other locations.

As yet another alternative, the operation of the system is distributed across the various devices. For example, the camera 12 may measure and generate data rate values and send the data rate values to a storage device 18 for storage and to system management devices 16a-b for live event detection . The system management devices 16a-b may then be further arranged to retrieve historical data rate values stored in the storage device and to analyze historical data rate values for event detection.

The nature of the series of data rate values and data rate values that represent events that occur in a scene may include, for example, when the camera is pointing up or towards the road or the length of the path, When capturing a scene moving from left to right, it can be highly dependent on the perspective captured by the camera. Thus, the camera's viewpoint can indicate the type of events that can be detected.

The above examples are all provided with respect to a fixed camera, but the camera may be a mobile camera. For example, the camera can be mounted on a drone or UAV (Unmanned Aerial Vehicle) or used as a dashboard camera on a vehicle.

Event detection can also be used in systems in which privacy masks are arranged to protect people's privacy. The event detector in accordance with the present invention can analyze the entire scene to generate events without providing any personally identifiable information.

Claims (15)

A method for detecting events,
Repeatedly registering a value indicative of an amount of data per unit time in a data stream generated by an encoder that encodes video from a scene by temporal compression, the data set indicating a change in data rate over time through the registering step Registered - and;
Analyzing a representation of the data rate over time of the data set to find features of the representation; And
And detecting events based on the found features of the expression
A method for detecting events. The method according to claim 1,
The step of analyzing the expression may comprise:
Analyzing a height of a peak in a representation of the data rate over time of the data set, analyzing a derivative of a portion of the representation of the data rate over time of the data set, Analyzing the area under the plot portion of the representation of the data rate over time.
A method for detecting events. 3. The method according to claim 1 or 2,
Generating models representative of specific events identifiable in the representation of the data rate over time of the data set; And
And identifying specific events using these models.
A method for detecting events. The method of claim 3,
Wherein one or more of the models are formed by identifying and registering a combination of specific features of a portion of the data rate representation over time of the data set representing a particular event
A method for detecting events. 5. The method of claim 4,
The step of identifying a particular event comprises:
And identifying the event as the specific event if the registered representation of the data rate for the time of the data set is not diverted from the model by more than a predetermined amount
A method for detecting events. The method of claim 3,
Wherein the models are formed by training a neural network to identify specific events
A method for detecting events. The method according to claim 1,
Further comprising storing the data set in a database,
Analyzing the representation and detecting the events are performed on the data set stored in the database.
A method for detecting events. The method according to claim 1,
Further comprising transmitting the data set to a client over a network, wherein analyzing the representation and detecting the events are performed by the client.
A method for detecting events. The method according to claim 1,
Further comprising the step of determining the direction of movement of the object triggering the detection of the event by analyzing the peak gradient of the representation of the data rate over time of the data set
A method for detecting events. The method according to claim 1,
Further comprising the step of determining the velocity of the object that triggers the detection of the event by analyzing the slope of the peak of the representation of the data rate over time of the data set
A method for detecting events. The method according to claim 1,
Further comprising the step of determining the velocity of the object that triggers the detection of the event by analyzing the area under the plot portion of the representation of the data rate over time of the data set
A method for detecting events. The method according to claim 1,
Further comprising the step of determining the size of the object that triggers the detection of the event by analyzing the area under the plot portion of the representation of the data rate over time of the data set
A method for detecting events. The method according to claim 1,
Wherein values representing data comprising image frames that are not temporally encoded are not included in the data set
A method for detecting events. The method according to claim 1,
Further comprising incrementing the counter each time an occurrence of an event is detected
A method for detecting events. The method of claim 3,
Further comprising incrementing a counter associated with a particular event whenever an occurrence of the particular event is identified
A method for detecting events.

Images