WO2016201683A1

WO2016201683A1 - Cloud platform with multi camera synchronization

Info

Publication number: WO2016201683A1
Application number: PCT/CN2015/081836
Authority: WO
Inventors: Song CAO; Genquan DUAN
Original assignee: Wizr
Priority date: 2015-06-18
Filing date: 2015-06-18
Publication date: 2016-12-22
Also published as: EP3311334A1; EP3311334A4

Abstract

The current invention discloses a method comprising: determining a first feature description of an event in a first video; determining a first location of the event in the first video; determining a first time stamp of the first video; determining a second feature description of an event in a second video; determining a second location of the event in the second video; determining a second time stamp of the second video; and determining a correlation value based on the first location, the second location, the first time stamp, the second time stamp, the first feature description, and the second feature description.

Description

CLOUD PLATFORM WITH MULTI CAMERA SYNCHRONIZATION

TECHNICAL FIELD

The invention is related to a video processing system.

SUMMARY OF INVENTION

The present invention relates to a system described in the disclosure and/or shown in the drawings.

The present invention relates to a method as described in the disclosure and/or shown in the drawings.

The present invention relates to a method comprising: determining a first feature description of an event in a first video； determining a first location of the event in the first video； determining a first time stamp of the first video； determining a second feature description of an event in a second video； determining a second location of the event in the second video； determining a second time stamp of the second video； and determining a correlation value based on the first location, the second location, the first time stamp, the second time stamp, the first feature description, and the second feature description.

The present invention relates to a video processing system comprising: one or more video storage locations storing a plurality of videos recorded from a plurality of video cameras； one or more video processors communicatively coupled with the one or more video storage locations, the one or more video processors configured to: determine a first feature description of an event in a first video； determine a first location of the event in the first video； determine a first time stamp of the first video； determine a second feature description of an event in a second video； determine a second location of the event in the second video； determine a second time stamp of the second video； and determine a correlation value based on the first location, the second location, the first time stamp, the second time stamp, the first feature description, and the second feature description.

The present invention relates to a method comprising: segmenting a video spanning a first duration of scene into a plurality of video clips having a specific time period； creating a visual dictionary of the scene from the plurality of video clips； determining features within the video clips； ranking the video clips based on the features； and creating a summarization video from the highest ranked video clips having a second duration that is significantly shorter than the first duration.

The present invention relates to a method comprising: track an event of interest in a specific number of frames within a video； determine the location of the event within each of the specific number of frames； and estimate a location of the event in a subsequent frame within the video based on the location of the event within each of the specific number of frames.

The present invention relates to a method comprising: determining that an event has occurred in a video； determining event features； determining whether the event is a false alarm event based on a comparison of the event features with event features in a false alarm database； and notifying a user about the event.

The present invention relates to a method comprising: determining that an event has occurred in a video； determining event features； determining whether the event is a false alarm event based on a comparison of the event features with event features in a false alarm database； notifying a user about the event； receiving an indication from the user that the event is a false alarm； and updating the false alarm database with the event features using a machine learning algorithm.

The present invention relates to a method for video processing comprising: converting the video to a second video that has at least one of a lower resolution and a lower frame rate； and processing the video.

The present invention relates to a method comprising: determining features and/or events in a video； determining foreground regions of at least a portion of the video that includes the features and/or events； determining a region of interest within one or more frames of the at least a portion of the video； and detecting the presence of a human within the region of interest within the at least a portion of the video.

The present invention relates to a method comprising: determining features and/or events in a video； determining foreground regions of at least a portion of the video that includes the features and/or events； determining a region of interest within one or more frames of the at least a portion of the video； detecting the presence of a human within the region of interest within the at least a portion of the video； and determining whether the features and/or events represent a false alarm.

BRIEF DESCRIPTION OF ACCOMPANYING DRAWING

These and other features, aspects, and advantages of the present disclosure are better understood when the following Detailed Description is read with reference to the accompanying drawings.

Figure 1 illustrates a block diagram of a system 100 for multi-camera synchronization.

Figure 2 is a flowchart of an example process for determining a similarity between two events recorded by one or more cameras according to some embodiments.

Figure 3 is a flowchart of an example process for summarizing video from a camera according to some embodiments.

Figure 4 is a flowchart of an example process for predicting the future location of an event in video frame and/or outputting a control value to a camera to track the event according to some embodiments.

Figure 5 shows an illustrative computational system for performing functionality to facilitate implementation of embodiments described herein.

Figure 6 is a flowchart of an example process for processing videos according to some embodiments.

Figure 7 is a flowchart of an example process for event filtering according to some embodiments.

DESCRIPTION OF PREFERRED EMBODIMENT

Systems and methods are disclosed for multi-camera synchronization of events. Systems and methods are also disclosed for creating summarization videos. Systems and methods are also disclosed for predicting the future location of events within a video frame.

Figure 1 illustrates a block diagram of a system 100 that may be used in various embodiments. The system 100 may include a plurality of cameras: camera 120, camera 121, and camera 122. While three cameras are shown, any number of cameras may be included. These cameras may include any type of video camera such as, for example, a wireless video camera, a black and white video camera, surveillance video camera, portable cameras, battery powered cameras, CCTV cameras, Wi-Fi enabled cameras, smartphones, smart devices, tablets, computers, GoPro cameras, wearable cameras, etc. The cameras may be positioned anywhere such as, for example, within the same geographic location, in separate geographic location, positioned to record portions of the same scene, positioned to record different portions of the same scene, etc. In some embodiments, the cameras may be owned and/or operated by different users, organizations, companies, entities, etc.

The cameras may be coupled with the network 115. The network 115 may, for example, include the Internet, a telephonic network, a wireless telephone network, a 3G network, etc. In some embodiments, the network may include multiple networks, connections, servers, switches, routers, connections, etc. that may enable the transfer of data. In some embodiments, the network 115 may be or may include the Internet. In some embodiments, the network may include one or more LAN, WAN, WLAN, MAN, SAN, PAN, EPN, and/or VPN.

In some embodiments, one more of the cameras may be coupled with a base station, digital video recorder, or a controller that is then coupled with the network 115.

The system 100 may also include video data storage 105 and/or a video processor 110. In some embodiments, the video data storage 105 and the video processor 110 may be coupled together via a dedicated communication channel that is separate than or part of the network 115. In some embodiments, the video data storage 105 and the video processor 110 may share data via the network 115. In some embodiments, the video data storage 105 and the video processor 110 may be part of the same system or systems.

In some embodiments, the video data storage 105 may include one or more remote or local data storage locations such as, for example, a cloud storage location, a remote storage location, etc.

In some embodiments, the video data storage 105 may store video files recorded by one or more of camera 120, camera 121, and camera 122. In some embodiments, the video files may be stored in any video format such as, for example, mpeg, avi, etc. In some embodiments, video files from the cameras may be transferred to the video data storage 105 using any data transfer protocol such as, for example, HTTP live streaming (HLS) , real time streaming protocol (RTSP) , Real Time Messaging Protocol (RTMP) , HTTP Dynamic Streaming (HDS) , Smooth Streaming, Dynamic Streaming over HTTP, HTML5, Shoutcast, etc.

In some embodiments, the video data storage 105 may store user identified event data reported by one or more individuals. The user identified event data may be used, for example, to train the video processor 110 to capture feature events.

In some embodiments, a video file may be recorded and stored in memory located at a user location prior to being transmitted to the video data storage 105. In some embodiments, a video file may be recorded by the camera and streamed directly to the video data storage 105.

In some embodiments, the video processor 110 may include one or more local and/or remote servers that may be used to perform data processing on videos stored in the video data storage 105. In some embodiments, the video processor 110 may execute one more algorithms on one or more video files stored with the video storage location. In some embodiments, the video processor 110 may execute a plurality of algorithms in parallel on a plurality of video files stored within the video data storage 105. In some embodiments, the video processor 110 may include a plurality of processors (or servers) that each execute one or more algorithms on one or more video files stored in video data storage 105. In some embodiments, the video processor 110 may include one or more of the components of computational system 500 shown in Fig. 5.

Figure 2 is a flowchart of an example process 200 for determining a similarity between two events recorded by one or more cameras according to some embodiments. One or more steps of the process 200 may be implemented, in some embodiments, by one or more components of system 100 of Figure 1, such as video processor 110. Although illustrated as discrete blocks, various blocks may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the desired implementation.

Process 200 begins at block 205 where a set of videos stored in a storage location such as, for example, video data storage 105, may be monitored by one or more processors such as, for example, video processor 110. In some embodiments, the set of videos may be stored in different local and/or remote storage locations. The separate storage locations, for example, may include local storage locations and/or remote storage locations relative to the video processor 110 that can be accessed by the video processor 110 directly and/or through the network 115.

A video V_i may be stored in the storage location that was recorded by camera C_i, where i is a number between 1 and the total number of videos stored in the storage location and/or the number of cameras. The video V_i may include a plurality of events E_ij, where j represents the number of events within a given video V_i.

At block 210 it may be determined that an eventE_i1 has occurred in a video V_i recorded by camera C_i. In some embodiments, low level events may be detected in block 210 such as, for example, motion detection events. In some embodiments, a feature detection algorithm may be used to determine that event E_i1 has occurred. In some embodiments, the feature description can be determined using a low level detection algorithm. In some embodiments, block 210 may occur in conjunction with block 215.

An event may include any number or type of occurrences captured by a video camera and stored in a video file. An event may include, for example, a person moving through a scene, a car or an object moving through a scene, a particular face entering the scene, a face, a shadow, animals entering the scene, an automobile entering or leaving the scene, etc.

If an event has not been found as determined by block 210, then the process 200 returns to block 205. If an event has been found, then the process 200 proceeds to block 215.

At block 215 a feature description f_i1 can be determined for the event E_i1. The feature descriptionf_i1 may be determined using a feature detector algorithm such as, for example, SURG, SIFT, GLOH, HOG, Affine shape adaptation, Harris affine, Hessian affine, etc. In some embodiments, the feature description can be determined using a high level detection algorithm. Various other feature detector algorithms may be used. In some embodiments, the feature description f_i1may be saved in the video storage location such as, for example, as metadata associated with the video V_i.

At block 220 the locationl_i1 of the feature f_i1 may be determined. The locationl_i1 of the featuref_i1 may be determined in the scene or in the camera frame. The locationl_i1, for example, may be represented in pixels. In some embodiments, the feature f_i1 may cover a number of pixels within a scene, the location l_i1may then be determined from the center of the feature f_il. In some embodiments, the locationl_i1may be saved in the video storage location such as, for example, as metadata associated with the video V_i.

At block 225 a time stamp t_i1 of the feature may be determined. In some embodiments, the timestamp may be an absolute time relative to some standard time. In some embodiments, the time stamp t_i1may be saved in the video storage location such as, for example, as metadata associated with the video V_i.

At block 230 a similarity measure may be determined for the event E_i1 relative to another event E_i1. In some embodiments, the similarity measure may be determined from the following equation:

δ (E_i1, E_mn) ＝ω₁Σ (f_i1-f_mn) +ω₂·d (l_i1, l_mn) +ω₃ |t_i1-t_mn| ,

where ω₁, ω₂, and ω₃ are constants that may be fixed for each camera, 1≤m≤number of videos, 1≤n≤ number of features in video V_m, and d (l_i1, l_mn) represent the Manhattan distance between the location l_i1 and location l_mn. ω₁, for example, may be equal to the inverse of n: ω₁＝1/n. ω₂, for example, may be a value between 0 and 10. In a specific example, ω₂＝1. ω₃, for example, may be a value between 0 and 10. In a specific example, ω₃＝1.

At block 235 it can be determined whether the event E_i1 and the event E_mn are related. For example, the events may be considered to be correlated if the similarity between the events is less than a threshold: δ (E_i1, E_mn) ＜T； where T is a threshold value. For example, the threshold value T ＝ 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, etc. If the event E_i1 and the event E_mn are correlated the two events may capture the same occurrence from the two different cameras C_i and C_m

The process 200 may be repeated for each video in the storage location or for specific videos in the storage location.

In some embodiments a greedy method may be used to find events to be correlated. In some embodiments, block 235 may determine the similarity between two events if the two videos or portions of the two videos are captured within a time window and/or if the two videos or portions of the two videos capture scenes that are with a specific geographic region relative to one another. For example, if an event E₁₁ is found in a video file recorded by camera 120 and camera 121 is physically close to camera 120, and/or camera 121 recorded a video file that is temporally close to event E₁₁ then process 200 may correlateevent E₁₁ withevents recorded with the video file produced by camera 121.

Figure 3 is a flowchart of an example process 300 for summarizing video from a camera according to some embodiments. One or more steps of the process 300 may be implemented, in some embodiments, by one or more components of system 100 of Figure 1, such as video processor 110. Although illustrated as discrete blocks, various blocks may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the desired implementation.

Process 300 begins at block 305 where video such as, for example, a video file from a camera, is segmented into a plurality of clips. For example, one or more videos representing all the video recorded from a camera during a time period may be segmented into a plurality of clips of a short duration. In some embodiments, the time period may be forty-eight, thirty-six, twenty-four, twelve, six, one, etc. hours. In some embodiments, the short duration may be sixty, thirty fifteen, ten, five, one, etc. seconds. In some embodiments, segmented video clips of the short duration may be created as individual files. In some embodiments, segmented video clips may be created virtually by identifying the beginning and end points of virtual video clips within the original video files.

At block 310 a visual dictionary may be created from the plurality of video clips. In some embodiments, the video dictionary may be built using an unsupervised method such as, for example, using a k-means to cluster segment video clips. Each class center of the segmented video clips may be labeled as a concept in the visual dictionary.

The visual dictionary may identify the visual features within the plurality of video clips. The visual dictionary may include faces, unique faces, background features, objects, animals, movement trajectories, color schemes, shading, brightness, etc. In some embodiments, the visual dictionary may be created from the video clips or other video files scene recorded by the same camera. In some embodiments, the visual dictionary may identify features and/or occurrences that differ from the background.

At block 315 the video clip may be converted into features using the visual dictionary. The video dictionary, for example, may represent a plurality of video concepts. These concepts may include, for example, features, actions, events, etc, such as, for example, walking, running, jumping, standing, automobile, color, animals, numbers, names, faces, etc. The video clip may be compared with the video clip and a concept confidence score may be associated with the video clip based on the correlation or comparison of the video clip with concepts in the video dictionary. These confidence scores may be associated with the video clip as descriptions and/or features of the video clip.

At block 320 the video clips may be ranked based on the features. For example, video clips with more unique features may be ranked higher than video clips with fewer unique features. As another example, video clips showing the background scene or mostly the background scene may be ranked lower. In some embodiments, video clips may be sorted based on the number of features or unique features in each video clip.

At block 325 a summarization video clip may be created from the highest ranked video clips. In some embodiments, the summarization video clip may have a specific length such as, for example, five, two, one, etc. minute. The summarization video clip may be created, for example with portions of each video clip ranked above a certain amount or including specific number of features. As another example, the summarization video clip may be created from a specific number of video clips. For example, for a one minute summarization video clip the top fifteen highest ranked video clips may be used. As another example, for a one minute summarization video clip the seven highest ranked video clips may be used.

Figure 4 is a flowchart of an example process 400 for predicting the future location of an event in video frame and/or outputting a control value to a camera to track the event according to some embodiments. One or more steps of the process 400 may be implemented, in some embodiments, by one or more components of system 100 of Figure 1, such as video processor 110, camera 120, camera 121, and/or camera 122. Although illustrated as discrete blocks, various blocks may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the desired implementation.

Process 400 begins at block 405 where an event (or feature) is tracked through a specific number of frames. An event can be determined, for example, in a manner similar to what is described in conjunction with block 210 or block 215 of process 200 shown in Figure 2. The specific number of frames, for example, for example, may include 2, 3, 4, 5, 6, 8, 10, 12, 15, 20, 25, or 30 frames.

At block 410 the location of the event may be determined in at least some of the specific number of frames. The location of the event can be determined, for example, in a manner similar to what is described in conjunction with block 220 of process 200 shown in Figure 2. The location of the event, for example, may include a vertical position, a horizontal position, a width, and/or a height of the event within a frame in pixel dimensions. The location may be written as r_i, j＝ (x, y, w, h) , where i represents the frame, j represents the camera, x represents the horizontal location of the event, y represents the vertical location of the event, w represents the width of the event, and h represents the height of the event.

At block 415 the location of the event in a subsequent frame such as, for example, frame i+n, where n is any number greater than 1, may be estimated. The location of the event in the subsequent frame may be determined based on the trajectory of the event in the specific number of frames. For example, the rate of change of the event through the previous specific number of frames can be estimated and then used to estimate the location of the event in the subsequent frame. Various other techniques may be used to track events.

At block 420 one or more shift values between the event location and the center of the frame in the subsequent frames can be determined based on the estimate found in block 415. At block 425 a camera instruction may be output that specifies a camera movement that centers the event in subsequent frames. In response, for example, the camera may be moved to center the event in the frame.

In some embodiments, there may be a delay between estimating the event in a subsequent frame and the response of the camera to a movement instruction. In such embodiments, the estimate of the location of the event in a subsequent frame may be made for a frame corresponding to the delay. For example, if the delay is five frames, then estimate may be made for an additional frame plus the delay frames: i+6 frames.

In some embodiments, r_i+n, j can be compared with estimated r_i+n, j. If r_i+n, j and estimated r_i+n, j are found to be similar then in future frames a region of interested defined by r_i+n, j can be determined and subsequent feature detections can occur within this region of interest. In some embodiments, the region of interest may include a buffer of a few pixels surrounding r_i+n, j.

In some embodiments, video processing may be sped up by decreasing the data size of a video. For example, a video may be converted into a second video by compressing the video, decreasing the resolution of the video, lower the frame rate, or some combination of these. For example, a video with a 20 frame per second frame rate may be converted to a video with a 2 frame per second frame rate. As another example, an uncompressed video may be compressed using any number of video compression techniques.

The computational system 500 (or processing unit) illustrated in Figure 5 can be used to perform and/or control operation of any of the embodiments described herein. For example, the computational system 500 can be used alone or in conjunction with other components. As another example, the computational system 500 can be used to perform any calculation, solve any equation, perform any identification, and/or make any determination described here.

The computational system 500 may include any or all of the hardware elements shown in the figure and described herein. The computational system 500 may include hardware elements that can be electrically coupled via a bus 505 (or may otherwise be in communication, as appropriate) . The hardware elements can include one or more processors 510, including, without limitation, one or more general-purpose processors and/or one or more special-purpose processors (such as digital signal processing chips, graphics acceleration chips, and/or the like) ； one or more input devices 515, which can include, without limitation, a mouse, a keyboard, and/or the like； and one or more output devices 520, which can include, without limitation, a display device, a printer, and/or the like.

The computational system 500 may further include (and/or be in communication with) one or more storage devices 525, which can include, without limitation, local and/or network-accessible storage and/or can include, without limitation, a disk drive, a drive array, an optical storage device, a solid-state storage device, such as random access memory ( “RAM” ) and/or read-only memory ( “ROM” ) , which can be programmable, flash-updateable, and/or the like. The computational system 500 might also include a communications subsystem 530, which can include, without limitation, a modem, a network card (wireless or wired) , an infrared communication device, a wireless communication device, and/or chipset (such as a

device, a 802.6 device, a Wi-Fi device, a WiMAX device, cellular communication facilities, etc. ) , and/or the like. The communications subsystem 530 may permit data to be exchanged with a network (such as the network described below, to name one example) and/or any other devices described herein. In many embodiments, the computational system 500 will further include a working memory 535, which can include a RAM or ROM device, as described above.

The computational system 500 also can include software elements, shown as being currently located within the working memory 535, including an operating system 540 and/or other code, such as one or more application programs 545, which may include computer programs of the invention, and/or may be designed to implement methods of the invention and/or configure systems of the invention, as described herein. For example, one or more procedures described with respect to the method (s) discussed above might be implemented as code and/or instructions executable by a computer (and/or a processor within a computer) . A set of these instructions and/or codes might be stored on a computer-readable storage medium, such as the storage device (s) 525 described above.

In some cases, the storage medium might be incorporated within the computational system 500 or in communication with the computational system 500. In other embodiments, the storage medium might be separate from the computational system 500 (e.g., a removable medium, such as a compact disc, etc. ) , and/or provided in an installation package, such that the storage medium can be used to program a general-purpose computer with the instructions/code stored thereon. These instructions might take the form of executable code, which is executable by the computational system 500 and/or might take the form of source and/or installable code, which, upon compilation and/or installation on the computational system 500 (e.g. , using any of a variety of generally available compilers, installation programs, compression/decompression utilities, etc. ) , then takes the form of executable code.

Figure 6 is a flowchart of an example process 600 for processing videos according to some embodiments. One or more steps of the process 600 may be implemented, in some embodiments, by one or more components of system 100 of Figure 1, such as video processor 110. Although illustrated as discrete blocks, various blocks may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the desired implementation.

At block 605 in the user module a user may indicate foreground regions with in the scene recorded in a video. In some embodiments, the user may indicate multiple regions within the scene that are considered foreground regions. In some embodiments, the user may do this graphically by drawing foreground areas within a scene. For example, a user may create a window by clicking or touching two areas within one or more frames on the video. The region of interest for a video frame may then be segmented into regions of interest using any process known in the art.

In some embodiments, an algorithm such as, for example, the grabcut algorithm may be used to segment the region of interest. For example, the algorithm may estimate the color distribution of the target object identified within the bounding box and/or that of the background using a Gaussian mixture model. The color distribution of the target object may then be used to construct a Markov random field over the pixel labels, with an energy function that prefers connected regions having the same label, and running a graph cut based optimization to infer their values. This process may be repeated a number of times until convergence. The result may provide a mask that blocks out the background.

In some embodiments, at block 605, human potential size may be determined by placing a graphical image of a human within the scene. The graphical image of the human may be scaled by the user until the human is approximately the size of a human recorded within the scene. Based on the amount of scaling, a typical human size may be determined. This may be repeated multiple times throughout the scene.

At block 610 a background model may be created from the first number of frames using the images mean and/or covariance. The background model may be created, for example, using a Gaussian Mixture Model (GMM) . The input, for example, can be a single frame from the video. The output, for example, can be a motion probability and/or a motion mask.

The background model may be initialized as M. For each input frame, f, the difference between f and the background model M can be calculated. Three Gaussians, g₁, g₂, g₃, can be used describe the three color channels of the input frame. Since each gaussian will output a probability, the probability of a mixture Gaussian is the sum of these three probabilities: p (i, j) ＝g₁+g₂+g₃, which can describe a region or pixel (i, j) as foreground or background. Each Gaussian may be described as g (I, μ, σ) ； where I represents the image, μ represents the mean, and σ represents the covariance.

In some embodiments, the whole background model is an online updating model. The model can keep updating the background parameter to handle change of viewpoint or illumination such as, for example, as the time of day changes and/or as the camera moves, pans, and/or tilts. In some embodiments, μand σ can be updated using a maximum likelihood algorithm such as, for example, an EM algorithm, when a new frame (s) is analyzed and/or added to the model.

In some embodiments, object tracking through the video frame can be performed by a tracking module at block 615. In some embodiments, five frames (or any number of frames) can be input into the tracking module and a predicted candidate region may be output. In some embodiments, the object tracking module can be used to determine, estimate, and/or predict regions in a frame representing a human (a candidate region) .

In some embodiments, the detection area can be limited. Assuming we have the detection result for frame f_i, where i＝1, . . . , 5 as r_ij, the recent detection responses may be stored in a buffer. Each detection response may be represented as r_ij＝ (x, y, w, h) . If the detection result r_ij and r_i'j' are similar, then there is a high probability that this is a region of interest. In some embodiments, an object detector may only be used in the predicted region.

At block 620 one or more humans may be detected in a video frame of the video. In some embodiments, a detection module may be used. In some embodiments, a frame may be input into the detection module with a region of interest defined. A detection response may be output. In some embodiments, the frame may be converted into a grayscale image. Associated Pairing Comparison Features (APCF) may be extracted from the grayscale image. The APCF may pare comparison of color and gradient orientation in granular space. The APCF feature may provide a chain of weak classifiers. Each classifier may provide a determination for the current extracted APCF feature such as, for example, the current region.

For example, when locating a human in a frame or image, APCF features in the whole image can be extracted. After this, a sliding window can scan the whole frame. Assume we already trained a detector M, in each sliding window, a chain of weak classifier will be used to determine if this area is a human or not. In detail, each sliding window needs to pass every weak classifier in order to decide this is the human.

At block 625 false alarm events can be learned. In some embodiments, false alarm data may be input into a self-learning module. The false alarm data, for example, may be collected from user input regarding various inputs, videos, and/or features. For example, SURF feature (or any other feature detector) may be used to compute features of the candidate a_i. The feature may be computed to match with the candidate SURF features； for example, the test image can be compared with the false alarm image. If the matched points are over a threshold value, T, which indicates the new detect results are much similar to the sample in our false alarm, then candidate a_i can be labeled as a false alarm. In some embodiments, the threshold value, T, can be calculated by the ratio of matched points divided by the total points.

In some embodiments, video processing may be spread among a plurality of servers located in the cloud or on cloud computing process. For example, different aspects, steps, or blocks of a video processing algorithm may occur on a different server. Alternatively or additionally, video processing may be for different videos may occur at different servers in the cloud.

In some embodiments, each video frame of a video may include metadata. For example, the video may be processed for event and/or object detection. If an event or an object occurs within the video then metadata associated with the video may include details about the object or the event. The metadata may be saved with the video or as a standalone file. The metadata, for example, may include the time, the number of people in the scene, the height of one or more persons, the weight of one or more persons, the number of cars in the scene, the color of one or more cars in the scene, the license plate of one or more cars in the scene, the identity of one or more persons in the scene, facial recognition data for one or more persons in the scene, object identifiers for various objects in the scene, the color of objects in the scene, the type of objects within the scene, the number of objects in the scene, the video quality, the lighting quality, the trajectory of an object in the scene, etc.

In some embodiments, feature detection may occur within a user specified region of interest. For example, a user may draw a bounding box in one or more frames of a scene recorded by a camera. The bounding box may define the region of interest within which events may be identified and/or tracked. In some embodiments, a user may specify two or more points within a frame and a polygonal shaped bounding box may be created based on these points. In some embodiments, the user may have the option to specify whether the bounding box bounds the region of interest or should be excluded from the region of interest.

Figure 7 is a flowchart of an example process 700 for event filtering according to some embodiments. One or more steps of the process 700 may be implemented, in some embodiments, by one or more components of system 100 of Figure 1, such as video processor 110. Although illustrated as discrete blocks, various blocks may be divided into T

The process 700 begins at block 705. At block 705 one or more videos may be monitored. In some embodiments, the videos may be monitored by a computer system such as, for example, video processor 110. In some embodiments, the videos may be monitored using one or more processes distributed across the Internet. In some embodiments, the one or more videos may include a video stream from a video camera or a video file stored in memory. In some embodiments, the one or more videos may have any file type.

At block 710 an event can be detected to have occurred in the one or more videos. The event may include a person moving through a scene, a car or an object moving through a scene, one or more faces being detected, a particular face leaving or entering the scene, a face, a shadow, animals entering the scene, an automobile entering or leaving the scene, etc. In some embodiments, the event may be detected using any number of algorithms such as, for example, SURG, SIFT, GLOH, HOG, Affine shape adaptation, Harris affine, Hessian affine, etc. In some embodiments, the event may be detected using a high level detection algorithm.

When an event is detected an event description may be created that includes various event data. The data may include data about the scene and/or data about objects in the scene such as, for example, object colors, object speed, object velocity, object vectors, object trajectories, object positions, object types, object characteristics, etc. In some embodiments, a detected object may be a person. In some embodiments, the event data may include data about the person such as, for example, the hair color, height, name, facial features, etc. The event data may include the time the event starts and the time the event stops. This data may be saved as metadata with the video.

In some embodiments, when an event is detected a new video clip may be created that includes the event. For example, the new video clip may include video from the start of the event to the end of the event.

In some embodiments, background and/or foreground filtering within the video may occur at some time during the execution of process 700.

If an event has been detected, then process 700 proceeds to block 715. If an event has not been detected, then process 700 returns to block 705.

At block 715 it can be determined whether the event is a false alarm event. In some embodiments, a false alarm event may be an event that has event data similar to event data in the false alarm database. The events data in the false alarm database may include data created using machine learning based on user input and/or other input. For example, the event data found in block 710 may be compared with data in the false alarm database.

If a false alarm event has been detected, then process 700 returns to block 705. If a false alarm event has been detected, then process 700 proceeds to block 725.

At block 725 a user may be notified. For example, the user may be notified using an electronic message such as, for example, a text message, an SMS message, a push notification, an alarm, a phone call, etc. In some embodiments, a push notification may be sent to a smart device (e.g., a smart phone, a tablet, a phablet, etc. ) . In response, an app executing on the smart device may notify the user that an event has occurred. In some embodiments, the notification may include event data describing the type of event. In some embodiments, the notification may also indicate the location where the event occurred or the camera that recorded the event.

At block 730 the user may be provided with an interface to indicate that the event was a false alarm. For example, an app executing on the users smart device (or an application executing on a computer) may present the user with the option to indicate that the event is a false alarm. For example, the app may present a video clip that includes the event to the user along with a button that would allow the user to indicate that the event is a false alarm. If a user indication has not been received, then process 700 returns to block 705. If a user indication has been received, then process 700 proceeds to block 735.

At block 735 the event data and/or the video clip including the event may be used to update the false alarm database and process 700 may then return to block 705. In some embodiments, machine learning techniques may be used to update the false alarm database. For example, machine learning techniques may be used in conjunction with the event data and/or the video clip to update the false alarm database. As another example, machine learning (or self-learning) algorithms may be used to add new false alarms to the database and/or eliminate redundant false alarms. Redundant false alarms, for example, may include false alarms associated with the same face, the same facial features, the same body size, the same color of a car, etc.

Process 700 may be used to filter any number of false alarms from any number of videos. For example, the one or more videos being monitored at block 705 may be a video stream of a door step scene (or any other location) . An event may be detected at block 710 when a person enters the scene. Event data may include data that indicates the position of the person in the scene, the size of the person, facial data, the time the event occurs, etc. The event data may include whether the face is recognized and/or the identity of the face. At block 720 it can be determined that the event is a false alarm when the facial data is compared with facial data in the false alarm database. If there is a match indicating that the face is known, then the event is a false alarm and process 700 returns back to block 705. Alternatively, if the facial data does not match facial data in the false alarm data, then process 700 moves to block 730 and an indication can be sent to the user, for example, through an app executing on their smartphone. The user can then visually determine whether the face is known by manually indicating as much through the user interface of the smartphone. The facial data may then be used to train the false alarm database.

In other examples process 700 may determine whether a car of specific make, model, color, and/or with certain license plates is a known car that has entered a scene and depending on the data in the false alarm database the user may be notified.

In other examples process 700 may determine whether an animal has entered a scene and depending on the data in the false alarm database the user may be notified.

In other examples process 700 may determine whether a person has entered the scene between specific hours.

In other examples process 700 may determine whether a certain number of people are found within a scene.

In some embodiments, video processing such as, for example, process 700, may be sped up by decreasing the data size of the video being processed. For example, a video may be converted into a second video by compressing the video, decreasing the resolution of the video, lower the frame rate, or some combination of these. For example, a video with a 70 frame per second frame rate may be converted to a video with a 7 frame per second frame rate. As another example, an uncompressed video may be compressed using any number of video compression techniques.

In some embodiments, at block 730 the user may indicate that the event is an important event that they would like to receive notifications about. For example, if the video shows a strange individual mulling about the users home during late hours, the user may indicate that they would like to be notified about such an event. This information may be used by the machine learning algorithm to ensure that such an event is not considered a false alarm and/or that the user is notified about the occurrence of such an event or a similar event in the future.

The term “substantially” means within 5％or 10％of the value referred to or within manufacturing tolerances.

Numerous specific details are set forth herein to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter may be practiced without these specific details. In other instances, methods, apparatuses, or systems that would be known by one of ordinary skill have not been described in detail so as not to obscure claimed subject matter.

Some portions are presented in terms of algorithms or symbolic representations of operations on data bits or binary digital signals stored within a computing system memory, such as a computer memory. These algorithmic descriptions or representations are examples of techniques used by those of ordinary skill in the data processing art to convey the substance of their work to others skilled in the art. An algorithm is a self-consistent sequence of operations or similar processing leading to a desired result. In this context, operations or processing involves physical manipulation of physical quantities. Typically, although not necessarily, such quantities may take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, or otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to such signals as bits, data, values, elements, symbols, characters, terms, numbers, numerals, or the like. It should be understood, however, that all of these and similar terms are to be associated with appropriate physical quantities and are merely convenient labels. Unless specifically stated otherwise, it is appreciated that throughout this specification discussions utilizing terms such as “processing, ” “computing, ” “calculating, ” “determining, ” and “identifying” or the like refer to actions or processes of a computing device, such as one or more computers or a similar electronic computing device or devices, that manipulate or transform data represented as physical, electronic, or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform.

The system or systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device can include any suitable arrangement of components that provides a result conditioned on one or more inputs. Suitable computing devices include multipurpose microprocessor-based computer systems accessing stored software that programs or configures the computing system from a general-purpose computing apparatus to a specialized computing apparatus implementing one or more embodiments of the present subject matter. Any suitable programming, scripting, or other type of language or combinations of languages may be used to implement the teachings contained herein in software to be used in programming or configuring a computing device.

Embodiments of the methods disclosed herein may be performed in the operation of such computing devices. The order of the blocks presented in the examples above can be varied—for example, blocks can be re-ordered, combined, and/or broken into sub-blocks. Certain blocks or processes can be performed in parallel.

The use of “adapted to” or “configured to” herein is meant as open and inclusive language that does not foreclose devices adapted to or configured to perform additional tasks or steps. Additionally, the use of “based on” is meant to be open and inclusive, in that a process, step, calculation, or other action “based on” one or more recited conditions or values may, in practice, be based on additional conditions or values beyond those recited. Headings, lists, and numbering included herein are for ease of explanation only and are not meant to be limiting.

While the present subject matter has been described in detail with respect to specific embodiments thereof, it will be appreciated that those skilled in the art, upon attaining an understanding of the foregoing, may readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, it should be understood that the present disclosure has been presented for-purposes of example rather than limitation, and does not preclude inclusion of such modifications, variations, and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art.

Claims

A system described in the disclosure and/or shown in the drawings.
A method as described in the disclosure and/or shown in the drawings.
A method comprising:

determining a first feature description of an event in a first video；

determining a first location of the event in the first video；

determining a first time stamp of the first video；

determining a second feature description of an event in a second video；

determining a second location of the event in the second video；

determining a second time stamp of the second video； and

determining a correlation value based on the first location, the second location, the first time stamp, the second time stamp, the first feature description, and the second feature description.
The method according to claim 3, wherein the correlation value is determined from:

δ (E₁, E₂) ＝ω₁Σ (f₁-f₂) +ω₂·d (l₁, l₂) +ω₃ |t₁-t₂| ,

where ω₁ is a constant, ω₂ is a constant, and ω₃ is a constant, 1≤m≤number of videos, 1≤n≤number of features in video V_m, and d (l₁, l₂) represents the distance between the first distance and the second distance.
A video processing system comprising:

one or more video storage locations storing a plurality of videos recorded from a plurality of video cameras；

one or more video processors communicatively coupled with the one or more video storage locations, the one or more video processors configured to:

determine a first feature description of an event in a first video；

determine a first location of the event in the first video；

determine a first time stamp of the first video；

determine a second feature description of an event in a second video；

determine a second location of the event in the second video；

determine a second time stamp of the second video； and

determine a correlation value based on the first location, the second location, the first time stamp, the second time stamp, the first feature description, and the second feature description.
A method comprising:

segmenting a video spanning a first duration of scene into a plurality of video clips having a specific time period；

creating a visual dictionary of the scene from the plurality of video clips；

determining features within the video clips；

ranking the video clips based on the features； and

creating a summarization video from the highest ranked video clips having a second duration that is significantly shorter than the first duration.
The method according to claim 6, wherein the first duration is longer than twelve hours and the second duration is shorter than five minutes.
The method according to claim 6, wherein the specific time period comprises less than sixty seconds.
A method comprising:

track an event of interest in a specific number of frames within a video；

determine the location of the event within each of the specific number of frames； and

estimate a location of the event in a subsequent frame within the video based on the location of the event within each of the specific number of frames.
The method according to claim 9, further comprising:

determining a shift between the center of the frame and the estimated location of the event； and

outputting the shift to the camera.
A method comprising:

determining that an event has occurred in a video

determining event features；

determining whether the event is a false alarm event based on a comparison of the event features with event features in a false alarm database； and

notifying a user about the event.
A method comprising:

determining that an event has occurred in a video；

determining event features；

determining whether the event is a false alarm event based on a comparison of the event features with event features in a false alarm database；

notifying a user about the event；

receiving an indication from the user that the event is a false alarm； and

updating the false alarm database with the event features using a machine learning algorithm.
A method for video processing comprising:

converting the video to a second video that has at least one of a lower resolution and a lower frame rate； and

processing the video.
A method comprising:

determining features and/or events in a video；

determining foreground regions of at least a portion of the video that includes the features and/or events；

determining a region of interest within one or more frames of the at least a portion of the video； and

detecting the presence of a human within the region of interest within the at least a portion of the video.
A method comprising:

determining features and/or events in a video；

determining foreground regions of at least a portion of the video that includes the features and/or events；

determining a region of interest within one or more frames of the at least a portion of the video；

detecting the presence of a human within the region of interest within the at least a portion of the video； and

determining whether the features and/or events represent a false alarm.