WO2009139723A1 - Method and device for analyzing video signals generated by a moving camera - Google Patents
Method and device for analyzing video signals generated by a moving camera Download PDFInfo
- Publication number
- WO2009139723A1 WO2009139723A1 PCT/SG2008/000188 SG2008000188W WO2009139723A1 WO 2009139723 A1 WO2009139723 A1 WO 2009139723A1 SG 2008000188 W SG2008000188 W SG 2008000188W WO 2009139723 A1 WO2009139723 A1 WO 2009139723A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- pixels
- frame
- shifting
- changes
- group
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Definitions
- the present invention relates in general to the field of image processing, and in particular to the analysis of video signals which are generated by a moving camera.
- GB 200507525 discloses a security monitoring system that uses video inputs to distinguish objects in motion from stationary objects.
- the system may set an alarm.
- the video stream is first processed on a frame by frame basis. Each frame is subject to edge detection processing, and then, groups of consecutive frames are compared to determine which of the detected edges persist from frame to frame. Any edges that do not persist are discarded. This removes data related to moving objects in the scene, such as people.
- US 2008002771 discloses a video segment which is analyzed to determine if it displays a scene that is stationary or has motion. When the video segment displays a scene with motion, the segment is further analyzed to determine if the motion resulted from camera movement, or from movement of the object that has been captured.
- This disclosure refers to two types of movements. The first one being a controlled movement, such as panning, tilting, zooming, rotation or forward or backward movement of the camera, while the second type is an unstable camera movement.
- both types of movements refer to certain camera movements that effect specific frames, this disclosure is far from providing a solution to the problem of extracting or identifying a moving object by analyzing a video steam created by a moving camera.
- a method for detecting one or more moving objects within a video signal generated by a moving camera comprising:
- moving camera as used herein and throughout the specification and claims, is used to denote a camera where there is a relative movement between the camera and the area/background being captured in the frame.
- the relative movement can be either due to the motion of the camera relatively to the area or due to the motion of the area relatively to the camera. Therefore, any reference to movement of the camera such as when the background shifting is described as being caused by the camera movement, any such reference should be understood that it may be interpreted as referring to the relative movement between the camera and the background.
- the camera being the entity that moves relatively to the background.
- shifting intensity value is used to denote the value of a parameter assigned to one or more pixels which location(s) in two different frames is/are known.
- the value of the shifting intensity indicates the change in the pixel(s) location(s) between the two frames.
- the changes determined in step (iii) had originated from a movement of the one or more objects in the area captured by the moving camera.
- the at least a second plurality of pixels is essentially identical to the first plurality of pixels.
- the method provided further comprises repeating steps (ii) to (vi) and wherein said first and second pluralities of pixels are associated with frames that respectively precede said first and second frames, enabling to detect the one or more moving objects within the video signal.
- the method provided further comprises a step of predicting shifting of background pixels and/or movement of said one or more moving objects within future frames, based on information derived from a present frame and its one or more preceding frames.
- the method provided by the present invention further comprises a step of re-identifying the at least one connected component by repeating steps (vi) and (vii).
- the predicted shifting of background pixels and/or of the movement of the one or more moving objects in future frames is used in identifying the at least a second plurality of pixels from among the first plurality of pixels in the second frame. For example, by knowing the location of the one or more moving object in future frames it is possible to project that knowledge to preceding frame(s) when that future frame becomes the present frame.
- the method may further comprise the step of: classifying the one or more moving objects within the video signal based on a relative movement identified between the one group of pixels and the background shifting.
- the step of classifying the one or more objects may be carried out by comparing the movement of the one connected component with that of objects that are stationary relative to the background (i.e. the changes in these objects' location, are the same as the changes in the location of the background pixels), as there are cases where it is easier to classify a moving object when compared to a background object rather than conducting the comparison with background pixels.
- the at least one connected component comprises at least two groups of pixels
- the method provided further comprises the step of: classifying the one or more moving objects within the video signal according to a relative movement identified between the at least two groups of pixels. For example, when identifying a person waiving his hand while walking, the pixels comprising the hand may be one group, the body of that person another group, while the relative movement between the two groups (as the hand is moved differently than the body) may provide a better means to classify that connected component (the person).
- the detection of the moving objects is carried out essentially in a real time detection process.
- the video signal is a live signal, but as those skilled in the art may appreciate, the same method mutates mutandis may be implemented on any video signal.
- the method is adapted to receive data which comprises information related to the camera movement (e.g. its velocity and/or direction) and incorporate the received data in the analysis process.
- a computer-readable medium comprising instructions that perform a method, when executed by a processor, for establishing a computerized process for detecting one or more moving objects within a video signal generated by a moving camera, which comprises:
- the computer-readable medium comprising instructions that perform a method may be for example a CD embedding software so that when it is inserted in a computer and operated, enables the detection of the one or more moving objects.
- a computer program product comprising a computer useable medium having computer readable program code embodied therein for detecting one or more moving objects within a video signal generated by a moving camera, the computer program product comprising: (i) computer readable program code for causing the computer to receive a video signal comprising a plurality of consecutive frames; (ii) computer readable program code for causing the computer to select a first plurality of pixels comprised in one of the plurality of consecutive frames being a first frame, and from among the first plurality of pixels to identify at least a second plurality of pixels comprised in a preceding frame, being a second frame;
- Fig. 2 - illustrates a schematic representation of a video signal
- Fig. 3 - demonstrates the first and second plurality of pixels in the example illustrated in Fig. 2;
- Fig. 4 - illustrates another schematic representation of a video signal with complex movement.
- Fig. 1 refers to three scenarios demonstrating the application of the "moving camera” concept according to the present invention, in which:
- Fig. 1A illustrates two points of references 110 and 120.
- the first point of reference 110 is located in camera 112, which is a stationary surveillance camera as known in the prior art
- the other reference point, 120 is located in one of the fixed objects comprised in the area captured by the camera, in this example, fence 122.
- the area being captured further comprises a tree 124, a rock 126 and a walking person 128. Since in this example the relative movement between the two points of reference is zero, there is no relative movement between the camera and the background, hence this example is a prior art example which is not encompassed by the present invention.
- Point of reference 130 is located in camera 132 which is airborne on airplane 134
- point of reference 140 is located in one of the fixed objects comprised in the area captured by the camera, in the example in rock 142.
- the area being captured further comprises a tree 148, a walking person 146 and a driving car 144.
- this example there is a relative movement between the camera and the background which results from the movement of the camera, hence this case serves as an example of a moving camera which falls under the definition of a moving camera of the present invention, and the method described by the present invention allows detecting the two moving objects 144 and 146.
- Fig. 1C illustrates two points of references 150 and 160, wherein the first, 150 is located in camera 152 which is placed on a watching tower, and the latter, 160, is on a boat deck 166 which is included in the area captured by the camera.
- Point of reference 160 is located in one of the fixed objects on the deck, e.g., anchor 162.
- the area being captured further comprises two lifebuoys 164, and two walking people 168.
- there is relative movement between the two points of reference caused by the motion of the sailing boat, hence although the camera is position on top of the tower it should be considered as a moving camera encompassed by the present invention due to the relative motion existing between the camera and the boat.
- the moving objects i.e. the people 168 walking on the deck may be differentiated and be detected separately from the moving boat.
- a video signal which comprises a plurality of consecutive frames.
- the video signal of the Fig. 2 example comprises a plurality of N consecutive frames.
- Frames 210-217 illustrated in this Fig. are only some of the frames comprised in the video signal.
- the frames are taken consecutively and describe the time evolution of a falling ball (226).
- the frames are taken so that the time gap between each two frames is identical to any time gap occurred while taking any other two consecutive frames.
- the method should not be understood as being restricted to a given number N of frames, and as will be further discussed, the video signal may be a live broadcast where the value of N may change with time to include additional information to improve the accuracy of the analysis results.
- a first plurality of pixels comprised in one of the plurality of consecutive frames being a first frame is selected and a second plurality of pixels is identified from among the first plurality of pixels in a preceding frame, being a second frame.
- a shifting intensity value is calculated for one or more of the identified pixels based on the changes determined.
- the shifting intensity value is a parameter assigned for each identified pixel in the second plurality of pixels, and is calculated from the respective pixel's transition in its location.
- all pixels in the second plurality i.e. included in the grey area of Fig. 3B
- the pixels comprising ball (226) will have the same shifting intensity value, which in fact will be derived based on the velocity at which the camera is moved. Since the ball is the only object that moved, the pixels associated therewith will have different shifting intensity value.
- a vector is generated for one or more of the identified pixels.
- the vector is associated with the location of the one or more of the identified pixels, and its calculated shifting intensity value.
- a vector is generated, where such vectors comprise data of the respective pixel's location at the first frame, and its shifting intensity value that was calculated in the previous step.
- one (or more) connected component is identified.
- the example illustrated in Fig. 2 is a rather simple one in several aspects.
- the ball movement is homogenous and this object (the ball) does not include parts that move differently from one another. Therefore, this step in the present example includes identifying only one group of pixels associated with a connected component, as the ball pixels will be in one group.
- the moving object i.e. the ball, is detected by associating the one connected component therewith.
- steps 2 to 6 may be further repeated mutates mutandis as long as required.
- the process may be continued quite easily, where every time a new first frame is defined and its preceding frame is considered as being the second frame, for the process described above.
- the method provided further comprises a step of predicting the background shifting and the movement of the one or more moving object.
- the predicted background shifting can further be used as an indication of a change in the movement of the camera. For example, if the camera is mounted on a vehicle that moves in a straight direction and all of a sudden the vehicle takes a sharp turn, the actual background shifting will be substantially different from the predicted one, and based on this difference one may derive some conclusions regarding the camera movement. Certain changes in the camera movement cause changes in the moving objects (e.g. from having the front view of the moving object to having its side view). As explained above, the example illustrated in Fig.
- FIG. 4A in which the moving objects are more than one ball (as is the case in the example of Fig .2).
- Fig. 4A three frames (410-412) are demonstrated out of a complete video signal. In these frames, one may observe a road (425) a person (420) and a bird (430). The camera in this part of the video stream is moving to the right causing the background to shift to the left. In addition, the person is walking to the right while waving his hand and a bird is flying in the sky.
- step Vl one more time. In this step, at least one connected component is identified.
- the connected component is defined as one which comprises at least one group of pixels from among the second plurality of pixels and wherein a change in each of the pixels comprised in the at least one group of pixels is associated with a change in each of the remaining pixels of that at least one group of pixels, and wherein the pixels comprised in each of the at least one group has a distinctive shifting intensity value thereby indicating a movement of the connected component relative to background shifting caused by the movement of the camera.
- Fig.4A there are two moving objects (the person and the bird), and neither one of them moves homogenously, therefore in this step we identify connected components that will eventually comprise these two moving objects. Let us first focus on the bird (Fig. 4B). According to the shifting intensity value three connected components may be identified in the bird.
- Each of the connected components corresponds to the above definition as it comprises one or more groups of pixels, where a change in one pixel in a group indicates a change in the rest of the pixels associated with that group.
- the left wing is a connected component having two groups of pixels (431" and 431")
- the body has only one group of pixels
- the connected component of the right wing also comprises two such groups of pixels (433 1 and 433"). Similar analysis may be conducted for the walking person.
- the method provided further comprises a step of classifying the moving objects. Two very distinct classifications may easily be demonstrated.
- the first one is when the moving object comprises only one connected component, which comprises only one group of pixels. This is a case of a homogenous movement which could be the ball in the example of Fig .2 or a car, a truck, and the like.
- the second type of classification is when the moving object comprises one or more connected components, and at least one of these connected elements comprises at least two different groups of pixels, where this type is associated with a more complex movement such as a person walking, flying bird and the like. Obviously, the latter classification may be further classified by depending on relationship between the various groups, number of groups etc.
- the various embodiments of the present invention may be carried out for real time detection of one or more moving objects or for non-real time analysis of video streams.
- first and second frames referred to herein and through the specification and claims may be in fact non-consecutive frames, but instead either representative frames or certain chosen frames (whether the choice is made arbitrary or not), e.g. when the movement of the moving object is not a very rapid one.
- iterative analysis explained hereinbefore does not have to be restricted to the selection of the first frame so that all iterations are in respect of the first selected frame, and different frames may be used throughout the analysis.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Studio Devices (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
Description
Claims
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP08754025A EP2289045A1 (en) | 2008-05-16 | 2008-05-16 | Method and device for analyzing video signals generated by a moving camera |
CN2008801288362A CN102203828A (en) | 2008-05-16 | 2008-05-16 | Method and device for analyzing video signals generated by a moving camera |
PCT/SG2008/000188 WO2009139723A1 (en) | 2008-05-16 | 2008-05-16 | Method and device for analyzing video signals generated by a moving camera |
AU2008356238A AU2008356238A1 (en) | 2008-05-16 | 2008-05-16 | Method and device for analyzing video signals generated by a moving camera |
IL207770A IL207770A0 (en) | 2008-05-16 | 2010-08-24 | Method and device for analyzing video signals generated by a moving camera |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/SG2008/000188 WO2009139723A1 (en) | 2008-05-16 | 2008-05-16 | Method and device for analyzing video signals generated by a moving camera |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2009139723A1 true WO2009139723A1 (en) | 2009-11-19 |
WO2009139723A8 WO2009139723A8 (en) | 2010-01-14 |
Family
ID=40200908
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/SG2008/000188 WO2009139723A1 (en) | 2008-05-16 | 2008-05-16 | Method and device for analyzing video signals generated by a moving camera |
Country Status (5)
Country | Link |
---|---|
EP (1) | EP2289045A1 (en) |
CN (1) | CN102203828A (en) |
AU (1) | AU2008356238A1 (en) |
IL (1) | IL207770A0 (en) |
WO (1) | WO2009139723A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101859436A (en) * | 2010-06-09 | 2010-10-13 | 王巍 | Large-amplitude regular movement background intelligent analysis and control system |
EP3336660A3 (en) * | 2016-12-14 | 2018-10-10 | Immersion Corporation | Automatic haptic generation based on visual odometry |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101799876B (en) * | 2010-04-20 | 2011-12-14 | 王巍 | Video/audio intelligent analysis management control system |
CN111381357B (en) * | 2018-12-29 | 2021-07-20 | 中国科学院深圳先进技术研究院 | Image three-dimensional information extraction method, object imaging method, device and system |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2004038659A2 (en) * | 2002-10-21 | 2004-05-06 | Sarnoff Corporation | Method and system for performing surveillance |
US20040201706A1 (en) * | 2001-10-26 | 2004-10-14 | Katsutoshi Shimizu | Corrected image generating apparatus and corrected image generating program storage medium |
US20050104964A1 (en) * | 2001-10-22 | 2005-05-19 | Bovyrin Alexandr V. | Method and apparatus for background segmentation based on motion localization |
US20050225637A1 (en) * | 2004-04-13 | 2005-10-13 | Globaleye Network Intelligence Ltd. | Area monitoring |
US20060078162A1 (en) * | 2004-10-08 | 2006-04-13 | Dynapel, Systems, Inc. | System and method for stabilized single moving camera object tracking |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101022505A (en) * | 2007-03-23 | 2007-08-22 | 中国科学院光电技术研究所 | Mobile target in complex background automatic testing method and device |
-
2008
- 2008-05-16 AU AU2008356238A patent/AU2008356238A1/en not_active Abandoned
- 2008-05-16 WO PCT/SG2008/000188 patent/WO2009139723A1/en active Application Filing
- 2008-05-16 CN CN2008801288362A patent/CN102203828A/en active Pending
- 2008-05-16 EP EP08754025A patent/EP2289045A1/en not_active Withdrawn
-
2010
- 2010-08-24 IL IL207770A patent/IL207770A0/en unknown
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050104964A1 (en) * | 2001-10-22 | 2005-05-19 | Bovyrin Alexandr V. | Method and apparatus for background segmentation based on motion localization |
US20040201706A1 (en) * | 2001-10-26 | 2004-10-14 | Katsutoshi Shimizu | Corrected image generating apparatus and corrected image generating program storage medium |
WO2004038659A2 (en) * | 2002-10-21 | 2004-05-06 | Sarnoff Corporation | Method and system for performing surveillance |
US20050225637A1 (en) * | 2004-04-13 | 2005-10-13 | Globaleye Network Intelligence Ltd. | Area monitoring |
US20060078162A1 (en) * | 2004-10-08 | 2006-04-13 | Dynapel, Systems, Inc. | System and method for stabilized single moving camera object tracking |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101859436A (en) * | 2010-06-09 | 2010-10-13 | 王巍 | Large-amplitude regular movement background intelligent analysis and control system |
EP3336660A3 (en) * | 2016-12-14 | 2018-10-10 | Immersion Corporation | Automatic haptic generation based on visual odometry |
US10600290B2 (en) | 2016-12-14 | 2020-03-24 | Immersion Corporation | Automatic haptic generation based on visual odometry |
Also Published As
Publication number | Publication date |
---|---|
AU2008356238A1 (en) | 2009-11-19 |
IL207770A0 (en) | 2010-12-30 |
WO2009139723A8 (en) | 2010-01-14 |
EP2289045A1 (en) | 2011-03-02 |
CN102203828A (en) | 2011-09-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
TWI750498B (en) | Method and device for processing video stream | |
Zhang et al. | Wide-area crowd counting via ground-plane density maps and multi-view fusion cnns | |
CN103824070B (en) | A kind of rapid pedestrian detection method based on computer vision | |
EP2330557B1 (en) | Moving body detection method and moving body detection device | |
Lazaridis et al. | Abnormal behavior detection in crowded scenes using density heatmaps and optical flow | |
Wojek et al. | Monocular 3d scene understanding with explicit occlusion reasoning | |
US20160191795A1 (en) | Method and system for presenting panoramic surround view in vehicle | |
JP2016099941A (en) | System and program for estimating position of object | |
CN104378582A (en) | Intelligent video analysis system and method based on PTZ video camera cruising | |
US10853949B2 (en) | Image processing device | |
JP5832910B2 (en) | Image monitoring device | |
US20120155707A1 (en) | Image processing apparatus and method of processing image | |
KR101548639B1 (en) | Apparatus for tracking the objects in surveillance camera system and method thereof | |
KR101472674B1 (en) | Method and apparatus for video surveillance based on detecting abnormal behavior using extraction of trajectories from crowd in images | |
CN110222616B (en) | Pedestrian abnormal behavior detection method, image processing device and storage device | |
CN103391424A (en) | Method for analyzing object in image captured by monitoring camera and object analyzer | |
Selver et al. | Camera based driver support system for rail extraction using 2-D Gabor wavelet decompositions and morphological analysis | |
Minoura et al. | Crowd density forecasting by modeling patch-based dynamics | |
EP2289045A1 (en) | Method and device for analyzing video signals generated by a moving camera | |
Saif et al. | Real time vision based object detection from UAV aerial images: a conceptual framework | |
Burkert et al. | People tracking and trajectory interpretation in aerial image sequences | |
US11544926B2 (en) | Image processing apparatus, method of processing image, and storage medium | |
WO2012153868A1 (en) | Information processing device, information processing method and information processing program | |
US9183448B2 (en) | Approaching-object detector, approaching object detecting method, and recording medium storing its program | |
KR20140045834A (en) | Method and apparatus for monitoring video for estimating size of single object |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 200880128836.2 Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 08754025 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 207770 Country of ref document: IL Ref document number: 2008754025 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2008356238 Country of ref document: AU |
|
WWE | Wipo information: entry into national phase |
Ref document number: 5348/CHENP/2010 Country of ref document: IN |
|
ENP | Entry into the national phase |
Ref document number: 2008356238 Country of ref document: AU Date of ref document: 20080516 Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |