EP2553924A1 - Effortless navigation across cameras and cooperative control of cameras - Google Patents
Effortless navigation across cameras and cooperative control of camerasInfo
- Publication number
- EP2553924A1 EP2553924A1 EP11758722A EP11758722A EP2553924A1 EP 2553924 A1 EP2553924 A1 EP 2553924A1 EP 11758722 A EP11758722 A EP 11758722A EP 11758722 A EP11758722 A EP 11758722A EP 2553924 A1 EP2553924 A1 EP 2553924A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- cameras
- camera
- ptz
- sample points
- per
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B13/00—Burglar, theft or intruder alarms
- G08B13/18—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength
- G08B13/189—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems
- G08B13/194—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
- G08B13/196—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
- G08B13/19639—Details of the system layout
- G08B13/19645—Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B13/00—Burglar, theft or intruder alarms
- G08B13/18—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength
- G08B13/189—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems
- G08B13/194—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
- G08B13/196—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
- G08B13/19639—Details of the system layout
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/62—Control of parameters via user interfaces
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
- H04N7/185—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source from a mobile camera, e.g. for remote control
Definitions
- the present invention generally relates to physical security systems and more specifically to systems for controlling cameras in video surveillance systems.
- An ideal solution goes beyond simply tracking a suspect, and enables the operator to navigate freely in the environment, e.g. to move around and be generally aware of the current situation in a specific area.
- Another common need is to enable virtual visitors to navigate inside and participate in a virtual reproduction of a building, tradeshow, shopping center or city.
- the ideal solution must also work equally well in real-time situation and during investigations of events and video sequences that have been archived.
- the solution must work reliably across a wide range of environments including facilities with multiple floors (transitions between floors, inside/outside), navigation across city-blocks where cameras may be on roofs, walls, moving vehicles (Unmanned Aerial Vehicle, car, elevator, bus, train), and so on.
- Most video surveillance solutions offer a 2D map to users. Using a map can help to identify possible cameras of interest, but constantly switching attention to/from videos/map distracts operators and increases the chance to miss suspicious activity. When switching cameras, humans also tend to oversimplify the problem and rely on simple cues like geographic proximity, i.e.
- More complex video analytics methods try to separate the suspicious person or object from the rest of the movement, but all known techniques are unreliable in complex real-life scenario, e.g. large number of people walking in multiple directions in a possibly dynamic environment (snow, rain, smoke). For the time being at least, only humans can make intelligent decisions to follow a specific individual in a crowded scene.
- Tracking can also be performed by identifying individuals, e.g. through biometrics like facial recognition, RFID tags, and GPS sensors. These techniques all suffer from limitations. Facial recognition techniques require a good view of the face and no known method is perfect, so false negatives and false positives are very frequent even in ideal scenarios. RFID and GPS require extra hardware and often the cooperation of the individual being tracked. None of these solutions provide much control to the operator when he desires to navigate without actually tracking a specific individual, to simply be aware of nearby activity.
- Cooperative Control of Cameras A related challenge is the effective and intuitive monitoring of a large outdoor area. Monitoring a large outdoor area (e.g. dozens or hundreds of cameras surrounding a facility) is challenging because each camera only gives limited a point of view. Operators often suffer from a "tunnel effect" because they only see a small amount of information at a time.
- the Omnipresence 3D software application includes a 3D video fusion capability to display many real-time or archived videos realistically on top of a 3D map.
- 3D video draping This is sometimes also referred to as 3D video draping or 3D video projection.
- 3D map viewport For each pixel in a 3D map viewport, a calculation is made to identify which fixed, panoramic or PTZ cameras has the best view of that pixel, and the 3D fusion is performed according to the precise, continuously-updated position, direction and field-of-view ("FOV") of each camera.
- FOV field-of-view
- a simple solution consists in providing simple user control, e.g. having the user click on a location in the 3D map, and having the system identify and redirect one or a few PTZ cameras that can see that location.
- One popular security design approach consists in dividing the perimeter into perimeter segments, and assigning one camera on or near the perimeter to monitor that perimeter segment constantly. [0024] Occasionally, one or some of these cameras are broken, obscured (e.g. by cargo ship, rain, sun glare), disconnected or otherwise unusable. When this happens, there is a gap in the perimeter that can be exploited by burglars, illegal immigrants or drug traffickers.
- the ideal solution would monitor all cameras and within seconds, identify when a camera is tampered with or unusable. The system would then automatically identify one or more PTZ cameras that cover the gap(s) optimally.
- the principles of the present invention are generally embodied in a scoring method 120 that evaluates input parameters including a point 115, an optional direction 116 and a list of cameras 103 (see Fig. 1).
- the point 115 is typically a point located near where the suspect was last seen.
- the direction 116 corresponds to the direction desired by the operator and/or the approximate direction the suspect went toward.
- the scoring method 120 extrapolates the likely areas the suspect may have gone toward, and identifies which cameras 103 have an optimal or partial view of that extrapolated area, then the higher-level application may use this scoring information to perform state changes 130, e.g.:
- Figure 1 is a high-level overview diagram of the navigation method.
- Figure 2 is a screenshot of the Omnipresence 3D software, one exemplary embodiment of the present invention.
- Figure 3 is another screenshot of the Omnipresence 3D software where a direction has been specified by the user.
- Figures 4A and 4B are an example of software pseudo-code for the scoring method, in accordance with the principles of the present invention.
- a point 115 and an optional direction 116 must first be provided in the same space in which cameras 300 are specified.
- Many approaches are possible.
- a user can mouse-down on a 2D map and drag to specify a direction relative to that map.
- a user can click on a 3D map, a 3D picking method can identify (e.g. through raytracing or rasterization) the closest intersecting polygon in the 3D model and its local normal, and the screen-space direction can be specified and projected on the polygon to obtain a direction expressed in the same space as the 3D map.
- the point and optional direction may be specified in a different space, e.g. to improve usability or take advantage of visual information provided by one or multiple cameras.
- the point 115 and optional direction 116 are specified through the exemplary Fast Track user interface, consisting in four main steps.
- a The user moves the mouse cursor over a camera viewport (e.g. main camera viewport 210 or secondary camera viewports 205).
- a circular indicator 215 is displayed along the ground plane, centered on the point 220 specified in camera viewport 210.
- the indicator 215 is displayed with the proper position, orientation and perspective transformation so it appears to be part of the displayed video image.
- Accurate placement of the indicator 215 is achieved using the viewport's camera calibration information in addition to the 3D model data to calculate the 3D geometry of the video image displayed in the viewport. This requires both accurate 3D geometry and camera calibration. Another option would be to extrapolate the local normal from a depth-map (e.g. generated through stereo) or point cloud (e.g. generated by laser scanner).
- a depth-map e.g. generated through stereo
- point cloud e.g. generated by laser scanner
- the main steps to properly place and display the circular indicator 215 are:
- intersection point is the 3D world position corresponding to the clicked point. (This is commonly known as a 3D picking operation.)
- the image-space point 100 is transformed 110 into a point 115 expressed in the map's coordinate system (e.g. world-space).
- map's coordinate system e.g. world-space
- the user drags the mouse in the direction where the subject is headed.
- an arrow indicator between the clicked position and the current position is displayed (Direction specified in camera viewport 310).
- the equivalent direction specified in the map 320 is displayed as well.
- the arrow is displayed with the proper position, orientation and perspective transformation so it appears to be part of the displayed video. This also transforms the image-space direction 101 into a direction 116 expressed in the 3D model's coordinate system.
- the last step is changed to:
- iv. Draw the target as a quad extending from the clicked point's 3D position to the current mouse cursor 3D world position. If the current cursor's 3D world position is further away than a fixed limit of 1.5 times the radius of the circular indicator 215, then the quad is limited to that size. If the cursor is less than 0.5 times the radius of the circular indicator 215, it is inside the inner circle, and is not displayed. These numbers (1.5, 0.5) are arbitrary and may be adjusted automatically depending on other parameters, for instance scaled according to the viewport or screen resolution.
- the direction arrow is oriented to track the current mouse cursor position and becomes longer as the user moves the cursor farther away from the circular indicator.
- the direction of the arrow is the direction for which the best camera will be searched while the length of the arrow may be used as an indicator of the relative distance along which the search for an optimal camera will be performed. This length is referred to as the "direction magnitude" in the scoring method.
- the direction magnitude scales the search distance from between 15 meters, when the arrow is just outside the circular indicator to 50 meters, when the arrow at its longest length.
- the point 115 and optional direction 116 can also be estimated using background subtraction and optical flow techniques, similar to those used in PTZ camera auto-tracking. For instance, the point 115 is estimated by the centroid or center of mass of a set of contiguous pixels (blob) identified by background subtraction. The direction is estimated using discrete differentiation of the point 115 from previous frames, or by using optical flow techniques such as tracking features within the blob. This extends the traditional PTZ Auto-tracking capability to automatically switch from camera to camera with no need for user intervention.
- Multiple scoring methods can be implemented and selected according to parameters such as indoor/outdoor, type of facility, etc.
- An exemplary embodiment corresponding to version 1 of the exemplary Omnipresence 3D Fast Track scoring method is shown in Figure 4 in pseudo-code format and described in details next.
- the scoring method is applied to candidate cameras, i.e. nearby video surveillance cameras in the 3D scene. Each candidate camera is evaluated to receive a per-camera score. The list of candidate camera may be generated dynamically. For less than 500 cameras, a brute-force approach (i.e. treat all cameras are candidates) is typically fast enough. For very large scenes (e.g. entire cities consisting of tens of thousands of cameras), the search may be optimized using accelerated hierarchical data structures (e.g.
- the scoring may be performed by extrapolating weighted sample points along a specific distribution shape.
- sample points may be distributed linearly on a 3D line starting from the point 115 and following the selected direction 116.
- Each sample point has a weight, e.g. sample points closer to the point 115 may have five times more weight than points 10 m away, to favor cameras that can see where the suspect just came from and to minimize the risk of losing sight of the suspect.
- the default weight distribution reaches its maximum 5 m from the point 115 along the direction 116 (i.e.
- the user input is approximate (i.e. the point he clicked may be a few pixels off, and the direction may be 10 degrees off)
- more complex sample point distributions are preferable. For instance, 30 lines of sample points may be distributed in a fan pattern starting from point 115 and covering 30 degrees centered at the direction 116 and distributed uniformly along the floor's plane, with weights lightly favoring the center of the fan.
- This approach is more forgiving to user input imprecision. It particularly helps usability in scenes with long hallways and when the user Fast Tracks far away from the camera. Because of the awkward perspective, the circular indicator and arrow appear much smaller and distorted, making it harder to aim correctly. To improve usability, a minimum size may be enforced for the circular indicator and arrow.
- the point 115 typically lies on a ground floor (since it is located by intersection with the 3D model), e.g. the user clicks where the feet of the suspect were located. If the sample point distribution were on the floor as well, a cameras looking directly at the feet of the suspect would be favoured over one that look at his face, which is rarely the desired outcome. For that reason, the distribution is typically elevated by 1 to 2 m so the optimal camera is more likely to center on the hands and face instead of the feet.
- a circular pattern may be used, or as described later in the section on the path finding improvement, the samples may be distributed to exhaustively to explore all possible paths, until all paths are completely covered by optimal cameras or until a maximum number of optimal cameras have been identified.
- the best optimal camera is displayed in the Main Camera Viewport 210, and optimal cameras with lower scores are displayed in the Secondary Camera Viewports 205. This approach maximizes the probability that a suspect that goes out of sight of any camera will eventually reappear in the Main Camera Viewport 210 or one of the Secondary Camera Viewports 205.
- a visibility test is performed to confirm that the sample point can be seen without any obstructing geometry. If the sample is visible, the camera score is increased by a quality coefficient multiplied by the sample weight; otherwise it remains the same for that sample.
- the quality coefficient applied on the sample weight may depend on a number of factors that qualify how well that camera sees that sample point. For instance, a camera with a higher resolution may be preferable. Instead of simply using the resolution of the camera itself, a more accurate quality factor is the physical pixel size (PPS), i.e. the physical distance between the left and right side of a pixel for that camera around the sample point. This can be estimated by projecting the sample point back on the camera to identify which pixel it fits in, then calculating the difference in angle between the left and right side, e.g. using the horizontal resolution and FOV information from the intrinsic camera calibration for a fixed or PTZ camera, or using the more complex calibration formulas typically provided by the lens manufacturer for a panoramic camera.
- PPS physical pixel size
- the PPS can then be estimated using the distance from the camera to the sample point. This value can then be converted into part of the final score for that camera and sample point, e.g. by mapping it using a linear scale from the minimum acceptable resolution (e.g. pixel size is 30 cm, faces are barely visible) to a very high resolution (e.g. pixel size of 5 mm so the face is very easily recognized). Different factors can be summed together or combined using more complex calculations. [0053]
- the visibility test can be performed in a number of ways. Ray casting and ray tracing are suitable for small 2D and 3D maps. For large 3D models, more efficient techniques such as depth maps may become necessary for real-time performance.
- the scoring may continue until a specific number of optimal cameras have been identified or a maximum search distance (e.g. 100 m) has been reached.
- the final score associated with each candidate camera is a real number bounded by -100000 and the number of samples and line segments used for the method.
- a camera will have a zero or negative score if it is already the currently selected camera or it is too far to have a good view of the selected direction.
- Figures 4A and 4B detail the pseudo-code of an exemplary implementation of the scoring method.
- Many optional techniques can significantly improve the reliability of the Fast Track method. As introduced earlier, a simple path finding approach enables Fast Track to explore all possible paths leading from the point 115.
- Path finding may be performed using a navigation map.
- the navigation map consists in a network of navigation points connected by segments where it is possible to move to and from. Typically, each room and hallway will include one or a few navigation points centered or away from the walls. For instance, indoors, the navigation map describes all possible paths someone can use to walk through the facility. Outdoors, the navigation map may consist in connected streets, tunnels and walkways.
- the navigation map may be complete (i.e. covering all possible paths) or only cover the paths and corners that prove problematic with the simpler Fast Track method.
- the navigation map may be integrated in the Fast Track method in multiple ways.
- the preferred embodiment consists in changing the sample distribution so it follows the possible paths starting near the point 115.
- the closest navigation segment is identified either through a brute force approach (suitable for navigation maps with less than 1000 segments), or through a spatial acceleration structure.
- a visibility test may be performed from the point 115 to the two closest navigation point and optionally, some of the points in between. This visibility test is useful to ensure that an occluder, e.g. a wall, does not separate the navigation segment from the point 115.
- sample points are distributed following the paths, by traversing the navigation map in a breadth-first fashion. Sample points are interspersed, e.g. every 50 cm, and are weighted more the closer they are from the initial sample point so that candidate cameras are considered optimal when they minimize gaps where the suspect will not be visible.
- the segments that have been traversed so far are memorized (e.g. using temporary flags on the original navigation map, or by building a temporary traversal tree). This ensures that navigation segments are never traversed twice.
- each branch is explored and the weights restart at high value to favour camera(s) that can observe the intersection where the multiple branches are visible.
- Samples suppression technique Optionally, when an optimal candidate camera is identified, the scoring on the remaining candidate cameras is performed again from the initial sample point, this time suppressing (i.e. ignoring) all sample points which were visible from the optimal camera. This scoring and suppression technique can be repeated X times to identify the X+1 cameras that produce a superior coverage of most sample points (e.g. on different paths) in a complementary fashion.
- the navigation map can link together different floors along staircases and escalators, the indoor and outdoor through entrances and large windows.
- the map viewport 230 may then be updated accordingly.
- Navigation segments may be updated dynamically. For instance, the segment going through a door can be disabled when the door is locked. The segment going from an elevator shaft on a specific floor, to the entrance in front of the elevator, can be enabled only when the elevator is on or near that specific floor. These disable states may be ignored when the user specified a direction with a sufficiently high magnitude.
- the navigation points themselves may move to account for moving vehicles such as trains on tracks.
- a moving camera may also be taken in consideration in the transformation 110 to obtain a precise world-space point 115. Movement and dynamic events may be extracted from third-party software (e.g. control software for elevator or train station), follow a time-based schedule, or be tracked and/or estimated through GPS, RFID and visual localization methods like SLAM (htt :// www.robots . ox . ac .uk/ ⁇ pnewman/ VisualSpatial.htm).
- 2D The use of a 3D map, while desirable for advanced features, is not necessary. A set of 2D maps and 2D camera placements, along with navigation segments linking different floors, may work reliably. For increased reliability, calculation may still be performed in 3D, assuming default camera and wall heights.
- Every cameras may be performed by measuring physically the position and location of cameras relative to the surrounding physical environment, and replicating this measurement in the 3D model.
- the intrinsic camera parameters e.g. focal length based on PTZ zoom factor
- the extreme FOVs maximum and min zoom factor
- PTZ position and orientation may be calibrated by taking the average of several calibrations looking in different directions (e.g. origin, 90 degrees along X, and 90 degrees along Y).
- panoramic cameras position and orientation may be calibrated by first applying a "virtual PTZ" transformation and treating it as a real PTZ afterwards. Although this process takes time, it was shown to produce acceptable final precision in practice.
- Panoramic and other non-affine lenses Remarkably, the same approach work reliably with cameras that have unusual projection properties.
- standard fish-eye and Panomorph panoramic lens manufactured by ImmerVision
- the Fast Track method can control PTZs to optimize the coverage of the area of interest. Two complementary techniques follow.
- Optimal PTZ control constrained to presets A set of PTZ presets (preset position and zoom) may be assigned to each PTZ camera. Each of these presets may then be scored independently to find an optimal preset. Once one preset has been considered optimal, the other presets assigned to this camera must be ignored since the PTZ cannot point in two directions at the same time. Since PTZ cameras move at a finite speed (e.g. 1 to 3 seconds for an entire revolution) and the video is generally too blurry to be useful when they are in movement, the weights can be multiplied by a PTZ difference factor to favour small movements. The same apply to zoom.
- a finite speed e.g. 1 to 3 seconds for an entire revolution
- the video is generally too blurry to be useful when they are in movement
- the weights can be multiplied by a PTZ difference factor to favour small movements. The same apply to zoom.
- the current PTZ parameters In order for the Fast Track user interface to work as expected, the current PTZ parameters must be known with sufficient precision (e.g. a few degrees) when initiating the Fast Track command. Newer PTZ cameras offer programmatic interfaces to query the current parameters. Older PTZ cameras can be made to work using the character reading technique described in a separate U.S. Provisional Application No. 61/315,603, entitled "Reading Characters in Low-Resolution and/or Highly Compressed Video Sequences", and incorporated herein by reference. [0075] While the first PTZ control technique works reliably, it requires that a user or automated technique first creates a large number of presets, and it will never find the absolute optimal PTZ parameters that would have maximized the score for a given camera. The technique that follows finds a reasonable set of PTZ parameters that is close to optimum in practice.
- Unconstrained optimal PTZ control Conceptually, for a given PTZ camera, all scoring sample points are considered at once. A visibility test is performed on each sample point, as usual. All sample points that pass the visibility test are then considered to find an optimal "PTZ window", along with their weights.
- the PTZ window is a rectangle in angular space that will maximize the score, by including the highest-weight samples. Since a PTZ has a limited field of view (e.g. 40 degrees), not all points may be visible. It is also generally desirable to zoom on the most important part of the scene (i.e. the sample points with the highest weight), instead of simply zooming out to cover as many samples as possible.
- One simple way to compute a PTZ window that is close to optimal consists in identifying the highest weight sample, then adding samples that are within search distance (e.g. 15 m). The angular bounding box of the samples that are left is then computed, and enlarged by 10% to provide context (e.g. so the hallway the person came through is visible). The aspect ratio of the bounding box is then corrected by adding padding vertically or horizontally, while keeping the original box centered. This bounding box can then be converted into pan-tilt-zoom values assuming calibrated PTZ cameras with known FOV per zoom value.
- FOV adjustment for Unconstrained optimal PTZ control If the computed FOV is too wide (i.e. larger than the maximum FOV supported by the camera when it is zoomed out), a new PTZ window is generated by keeping one corner constant (e.g. top-left) and shifting the corner inwards so it fits exactly in the maximum FOV.
- Unconstrained optimal PTZ control combined with Improved Sample Suppression The Unconstrained optimal PTZ control method may be adjusted to account for suppressed samples by inserting the initial step of pre-multiplying the weight for each sample by the quality coefficient for this PTZ, subtracting (the best quality coefficient of a previously optimal camera * weight) and clamping to a minimum value of 0. Essentially, the samples that have the most importance are those with a high-original weight, that no optimal camera identified so far can see with high quality. During evaluation for this PTZ, the resolution-specific part of the quality coefficient can be assigned to the maximum zoom value for this PTZ.
- the scores for all samples are re-evaluated, this time taking in consideration the real zoom-value selected during the FOV adjustment for the quality coefficient calculation. This ensures that cameras that can zoom very far but are actually assigned a low zoom to widen the field of view, are not favoured over cameras that can zoom less far but still obtain a superior effective resolution after FOV adjustment.
- the scoring method can accept a point 115 that is not limited to a ground floor or a user-specified point, by simply eliminating the sample point projection step so only the point 115 is scored against.
- the same technique can readily handle complementary actions such as redirecting one or multiple cameras on a user-provided point on an arbitrary point in the map (e.g. the point directly at the center of the map, so PTZs continuously follow the movement of the map), or to continually track a target provided by a third-party system such as a ground-based radar or GPS across multiple cameras.
- Omnipresence 3D the flying navigation is by default always enabled and performed using the Space Navigator to navigate on the 3D map, while the Fast Track navigation is performed by left clicking on the map or in any Camera Viewport.
- Omnipresence 3D includes a 3D video fusion capability to display dozens of real-time or archived videos realistically on top of the 3D map. (This is sometimes referred to as 3D video draping or 3D video projection.) For each pixel in the 3D map viewport, a calculation is made to identify which fixed, panoramic or PTZ cameras has the best view of that pixel, and the 3D fusion is performed according to the precise, continuously-updated position, direction and FOV of each camera.
- a simple but non-optimized approach for 3D video fusion is described here summarily.
- the simplest implementation consists in rendering the 3D view with a large pixel shader (e.g. using Microsoft DirectX 10) that computes, for each pixel in the 3D view, which camera has the optimal view of that pixel.
- the visibility is processed using shadow maps computed every time the camera direction or field of view changes.
- the shadow map can use the same resolution as the actual camera resolution. This naive approach is relatively easy to implement and will provide an acceptable performance as long as the number of cameras on the screen is kept low, e.g. 3 cameras at a time.
- Cooperative Control of Cameras for an Optimal Coverage of the 3D Map (C3DM): This feature is a variation of the previously described methods. Instead of using a distribution centered on a 3D point 115 as in Fast Track, the distribution can be generalized to consist in a weighted set of samples points distributed over one or more areas of interest, i.e. areas that should be monitored by the cameras. Increasing weights indicate relatively higher importance or priority. For instance, to enable easy tracking of suspects outdoor using available cameras by moving in a 3D viewport, the sample point distribution should cover the entire area visible from that 3D viewport. The distribution may consist in a grid pattern projected from the 3D map viewport 230, intersecting with the underlying 3D model representing the terrain and buildings.
- the grid may consist in cells 10 pixels wide and 10 pixels large and the weights may be uniform over the entire grid (i.e. so that camera coverage is distributed evenly over the viewport, providing a better "big picture” view) or the weight distribution may favour the center point of the viewport, with a linear decrease from 1 in the center to 0.5 in the corners (i.e. the center of the 3D map viewport 230 is typically where a suspect will be when tracking in flying navigation).
- the samples are points on the 3D model that appear aligned in a grid when seen from the current 3D map viewport.
- the optimal cameras viewing directions may be roughly aligned with the viewing direction in the 3D map viewport 230.
- the quality coefficient may be extended to take in consideration the similitude in direction.
- the Improved Sample Suppression technique may then be applied X times to display X+l cameras with complementary views of the 3D map viewport. Optimal fixed and panoramic cameras may be scored and identified first, since they do not change orientation/zoom and. PTZ cameras are evaluated afterwards so they focus on the remaining sample points for which fixed and panoramic cameras had a relatively low quality coefficient.
- Self-Healing Perimeter SHP: This feature leverages one or multiple PTZ cameras (and variations like fixed cameras on pan-tilt heads) to fill one or multiple gaps in a security perimeter. It is assumed that gap(s) in the perimeter segments have been detected (e.g. through video analytics, tamper detection in the camera, etc.), and one or multiple PTZ cameras can be freely directed to close some of these gaps. SHP may then be implemented in several steps:
- sample point distribution that covers that segment. For instance, if the perimeter segment is defined as set of lines or curves in 2D or 3D, a cylinder distribution can be generated by extruding a circle with a radius of 2 meters with 16 sample points, with one circle every 2 meters along the fence.
- each sample point also gets a sample normal pointing outwards from the center of the circle, so that only sample points whose sample normal faces a given camera are considered. Otherwise many distribution points would not pass the visibility test when the fence is considered opaque (e.g. wall), and it would be hard to favour cameras that can see the entire perimeter segment from one side of the fence as opposed to half of the perimeter from both sides, leaving a gap of half of the perimeter segment unmonitored.
- the C3DM technique is applied to one perimeter segment at a time, with the following adaptations. Instead of using a grid from the 3D map viewport, one perimeter segment sample distribution is scored at a time, because it is typically preferable for the selected optimal camera(s) to have a limit FOV such that the perimeter segment is covered with maximum resolution, making it easier for a human to monitor individual segments or for video analytics to perform reliably.
- FVT Fracing Visibility Test
- the FVT for a given camera is computed by dividing the number of samples on that segment that are visible by the candidate camera, by the number of samples on that segment that are facing the candidate camera, i.e. positive dot product between (sample point position - camera position) and sample normal).
- An appropriate FVT threshold may differ based on the desired security redundancy and availability of spare PTZ cameras. For a facility with critical perimeter security needs and plenty of PTZ cameras (or perimeter cameras on pan-tilt heads), a good default percentage may be 75%. Facilities with less stringent security needs may lower this value, e.g. to 33% to at least achieve a good partial coverage of each gap.
- the FVT may be combined with a per-sample, per-camera resolution test, e.g. by specifying a Minimum Physical Pixel Size ("MPPS") for each segment.
- MPPS Minimum Physical Pixel Size
- segments on the sea could specify a MPPS of 50 cm (since the objects of interests are big, e.g. boats), while the MPPS on segments where human are expected to pass through (e.g. gate) may specify a MPPS of 1 cm to provide enough resolution to roughly recognize the person coming through.
- the MPPS could also be automatically derived from the physical length of perimeter segments, e.g. by dividing the length of each segment by a typical horizontal resolution like 720 or 1024 pixels, and by dividing by a conservative factor (e.g. the FVT percentage) so a less-than-perfect camera placement may still pass.
- a conservative factor e.g. the FVT percentage
- Any optimal camera that passes both the FVT and MPPS tests is considered a good match for that segment, and other candidate cameras are considered a bad match.
- the process can be applied to pairs and other combinations of candidate cameras. Instead of evaluating the FVT and MPPS test on a candidate camera at a time, for each sample, the tests are considered to pass if they both pass on one of the candidate cameras. This way pairs of cameras that complement each other will result in a higher combined FVT score that may exceed the desired FVT threshold.
- a fixed or panoramic camera may cover multiple segments with acceptable FVT and MPPS, but in most cases where the perimeter is wide and far away, PTZ cameras will only be assigned to a single perimeter segment so once assigned, it is taken out of the list and remaining assignments are sorted accordingly.
- PTZ cameras that are assigned may be locked so users cannot accidentally or purposefully direct them somewhere where the gap would be re- opened. Cameras that are unassigned may be freely operated by the users. Further constraints and prioritizations may easily be implemented as well, e.g. so that PTZ cameras are excluded or favored over others.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Closed-Circuit Television Systems (AREA)
- Studio Devices (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US31805510P | 2010-03-26 | 2010-03-26 | |
PCT/CA2011/000319 WO2011116476A1 (en) | 2010-03-26 | 2011-03-25 | Effortless navigation across cameras and cooperative control of cameras |
Publications (3)
Publication Number | Publication Date |
---|---|
EP2553924A1 true EP2553924A1 (en) | 2013-02-06 |
EP2553924A4 EP2553924A4 (en) | 2014-01-15 |
EP2553924B1 EP2553924B1 (en) | 2017-05-10 |
Family
ID=44672415
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP11758722.0A Not-in-force EP2553924B1 (en) | 2010-03-26 | 2011-03-25 | Effortless navigation across cameras and cooperative control of cameras |
Country Status (8)
Country | Link |
---|---|
US (1) | US9544489B2 (en) |
EP (1) | EP2553924B1 (en) |
AU (1) | AU2011232280A1 (en) |
BR (1) | BR112012024415A2 (en) |
CA (1) | CA2794057C (en) |
MX (1) | MX2012011118A (en) |
SG (1) | SG184520A1 (en) |
WO (1) | WO2011116476A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108596033A (en) * | 2018-03-22 | 2018-09-28 | 南京邮电大学 | Quick holder localization method towards Intelligent household scene |
Families Citing this family (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2012174603A1 (en) * | 2011-06-24 | 2012-12-27 | Honeywell International Inc. | Systems and methods for presenting dvm system information |
US10362273B2 (en) | 2011-08-05 | 2019-07-23 | Honeywell International Inc. | Systems and methods for managing video data |
US20130201339A1 (en) * | 2012-02-08 | 2013-08-08 | Honeywell International Inc. | System and method of optimal video camera placement and configuration |
EP2645701A1 (en) * | 2012-03-29 | 2013-10-02 | Axis AB | Method for calibrating a camera |
US9367065B2 (en) | 2013-01-25 | 2016-06-14 | Google Inc. | Modifying behavior of autonomous vehicles based on sensor blind spots and limitations |
US9426451B2 (en) * | 2013-03-15 | 2016-08-23 | Digimarc Corporation | Cooperative photography |
US10277869B2 (en) * | 2014-12-11 | 2019-04-30 | Sensormatic Electronics, LLC | Efficient process for camera call-up |
US9984494B2 (en) * | 2015-01-26 | 2018-05-29 | Uber Technologies, Inc. | Map-like summary visualization of street-level distance data and panorama data |
US9842268B1 (en) * | 2015-03-27 | 2017-12-12 | Google Llc | Determining regions of interest based on user interaction |
US10269257B1 (en) * | 2015-08-11 | 2019-04-23 | Gopro, Inc. | Systems and methods for vehicle guidance |
US9652828B1 (en) | 2015-12-29 | 2017-05-16 | Motorola Solutions, Inc. | Method and apparatus for imaging a scene |
WO2017176550A1 (en) * | 2016-04-05 | 2017-10-12 | Pcms Holdings, Inc. | Method and system for autonomous vehicle sensor assisted selection of route with respect to dynamic route conditions |
EP3229174A1 (en) | 2016-04-06 | 2017-10-11 | L-1 Identity Solutions AG | Method for video investigation |
US10943088B2 (en) | 2017-06-14 | 2021-03-09 | Target Brands, Inc. | Volumetric modeling to identify image areas for pattern recognition |
US10921142B2 (en) | 2017-12-14 | 2021-02-16 | Waymo Llc | Methods and systems for sun-aware vehicle routing |
TWI662808B (en) * | 2018-03-16 | 2019-06-11 | 中強光電股份有限公司 | Remote management system and method |
US10931863B2 (en) | 2018-09-13 | 2021-02-23 | Genetec Inc. | Camera control system and method of controlling a set of cameras |
FR3088041B1 (en) * | 2018-11-02 | 2020-10-16 | Renault Sas | PROCESS FOR DEVELOPING A DRIVING INSTRUCTIONS FOR A MOTOR VEHICLE |
DE102019201490A1 (en) * | 2019-02-06 | 2020-08-06 | Robert Bosch Gmbh | Calibration device for a monitoring device, monitoring device for man-overboard monitoring and method for calibration |
US11516517B2 (en) | 2021-03-19 | 2022-11-29 | Sm Tamjid | Localized dynamic video streaming system |
WO2023277829A1 (en) * | 2021-07-01 | 2023-01-05 | Chinwatanakit Worapot | Position and time tracking system based on photographs |
CN114460862A (en) * | 2021-12-31 | 2022-05-10 | 北京瞰瞰智能科技有限公司 | Camera equipment ring test simulation method applied to ADAS |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060203090A1 (en) * | 2004-12-04 | 2006-09-14 | Proximex, Corporation | Video surveillance using stationary-dynamic camera assemblies for wide-area video surveillance and allow for selective focus-of-attention |
US20100020175A1 (en) * | 2008-07-24 | 2010-01-28 | Tomomi Takada | Video-recording and transfer apparatus, and video-recording and transfer method |
Family Cites Families (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5956081A (en) * | 1996-10-23 | 1999-09-21 | Katz; Barry | Surveillance system having graphic video integration controller and full motion video switcher |
US7522186B2 (en) * | 2000-03-07 | 2009-04-21 | L-3 Communications Corporation | Method and apparatus for providing immersive surveillance |
US20030123703A1 (en) * | 2001-06-29 | 2003-07-03 | Honeywell International Inc. | Method for monitoring a moving object and system regarding same |
US20050012817A1 (en) * | 2003-07-15 | 2005-01-20 | International Business Machines Corporation | Selective surveillance system with active sensor management policies |
US20080055101A1 (en) * | 2004-03-19 | 2008-03-06 | Intexact Technologies Limited | Location Tracking System And A Method Of Operating Same |
US7629995B2 (en) * | 2004-08-06 | 2009-12-08 | Sony Corporation | System and method for correlating camera views |
EP1872345B1 (en) * | 2005-03-25 | 2011-03-02 | Sensormatic Electronics, LLC | Intelligent camera selection and object tracking |
US7944454B2 (en) * | 2005-09-07 | 2011-05-17 | Fuji Xerox Co., Ltd. | System and method for user monitoring interface of 3-D video streams from multiple cameras |
US7884849B2 (en) * | 2005-09-26 | 2011-02-08 | Objectvideo, Inc. | Video surveillance system with omni-directional camera |
US8218826B2 (en) * | 2006-05-19 | 2012-07-10 | Schlumberger Technology Corporation | Integrated measurement based on an optical pattern-recognition |
US7667596B2 (en) * | 2007-02-16 | 2010-02-23 | Panasonic Corporation | Method and system for scoring surveillance system footage |
US8527655B2 (en) * | 2007-05-22 | 2013-09-03 | Vidsys, Inc. | Optimal routing of audio, video, and control data through heterogeneous networks |
US20100208064A1 (en) * | 2009-02-19 | 2010-08-19 | Panasonic Corporation | System and method for managing video storage on a video surveillance system |
-
2011
- 2011-03-25 EP EP11758722.0A patent/EP2553924B1/en not_active Not-in-force
- 2011-03-25 US US13/636,348 patent/US9544489B2/en active Active
- 2011-03-25 AU AU2011232280A patent/AU2011232280A1/en not_active Abandoned
- 2011-03-25 MX MX2012011118A patent/MX2012011118A/en active IP Right Grant
- 2011-03-25 CA CA2794057A patent/CA2794057C/en not_active Expired - Fee Related
- 2011-03-25 WO PCT/CA2011/000319 patent/WO2011116476A1/en active Application Filing
- 2011-03-25 SG SG2012074993A patent/SG184520A1/en unknown
- 2011-03-25 BR BR112012024415A patent/BR112012024415A2/en not_active IP Right Cessation
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060203090A1 (en) * | 2004-12-04 | 2006-09-14 | Proximex, Corporation | Video surveillance using stationary-dynamic camera assemblies for wide-area video surveillance and allow for selective focus-of-attention |
US20100020175A1 (en) * | 2008-07-24 | 2010-01-28 | Tomomi Takada | Video-recording and transfer apparatus, and video-recording and transfer method |
Non-Patent Citations (1)
Title |
---|
See also references of WO2011116476A1 * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108596033A (en) * | 2018-03-22 | 2018-09-28 | 南京邮电大学 | Quick holder localization method towards Intelligent household scene |
CN108596033B (en) * | 2018-03-22 | 2021-09-24 | 南京邮电大学 | Rapid cloud deck positioning method for intelligent home scene |
Also Published As
Publication number | Publication date |
---|---|
AU2011232280A1 (en) | 2012-11-08 |
US9544489B2 (en) | 2017-01-10 |
CA2794057A1 (en) | 2011-09-29 |
WO2011116476A1 (en) | 2011-09-29 |
EP2553924B1 (en) | 2017-05-10 |
SG184520A1 (en) | 2012-11-29 |
MX2012011118A (en) | 2013-04-03 |
CA2794057C (en) | 2019-07-23 |
EP2553924A4 (en) | 2014-01-15 |
BR112012024415A2 (en) | 2017-08-08 |
US20130010111A1 (en) | 2013-01-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9544489B2 (en) | Effortless navigation across cameras and cooperative control of cameras | |
Morse et al. | UAV video coverage quality maps and prioritized indexing for wilderness search and rescue | |
JP5153302B2 (en) | Computer-implemented method for measuring the performance of a monitoring system | |
US7796154B2 (en) | Automatic multiscale image acquisition from a steerable camera | |
US8624962B2 (en) | Systems and methods for simulating three-dimensional virtual interactions from two-dimensional camera images | |
US20070058717A1 (en) | Enhanced processing for scanning video | |
CN107836012A (en) | Mapping method between projection image generation method and its device, image pixel and depth value | |
US20050012817A1 (en) | Selective surveillance system with active sensor management policies | |
EP2830028B1 (en) | Controlling movement of a camera to autonomously track a mobile object | |
CA2466085A1 (en) | Method and apparatus for providing immersive surveillance | |
AU2021255130B2 (en) | Artificial intelligence and computer vision powered driving-performance assessment | |
KR102107189B1 (en) | System and method for maintenance history and state management of facilities of subway station onto 3D screen | |
CN110378292A (en) | Three dimension location system and method | |
Roth et al. | Next-generation 3D visualization for visual surveillance | |
KR101686797B1 (en) | Method for analyzing a visible area of a closed circuit television considering the three dimensional features | |
KR20200086561A (en) | System and method to be connected video data of CCTV camera system with 3D control system of subway station | |
JP6358996B2 (en) | Security simulation device | |
US11785196B2 (en) | Enhanced three dimensional visualization using artificial intelligence | |
Muraoka et al. | Chapter Localizing and Visualizing the Degree of People Crowding with an Omnidirectional Camera by Different Times | |
KR20210013238A (en) | System and method to be connected video data of CCTV camera system with 3D control system of subway station | |
Chen et al. | Immersive surveillance for total situational awareness | |
Kuman et al. | Three-dimensional omniview visualization of UGS: the battlefield with unattended video sensors | |
Watanabe et al. | A Walkthrough System to Display Video Corresponding to the Viewer's Face Orientation | |
Zheng et al. | View planning for cityscape archiving and visualization |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20121017 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
DAX | Request for extension of the european patent (deleted) | ||
A4 | Supplementary search report drawn up and despatched |
Effective date: 20131213 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: H04N 5/232 20060101ALI20131209BHEP Ipc: H04N 7/18 20060101AFI20131209BHEP Ipc: G08B 13/196 20060101ALI20131209BHEP |
|
17Q | First examination report despatched |
Effective date: 20160204 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
INTG | Intention to grant announced |
Effective date: 20161124 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 893430 Country of ref document: AT Kind code of ref document: T Effective date: 20170515 Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602011037840 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20170510 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 893430 Country of ref document: AT Kind code of ref document: T Effective date: 20170510 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170811 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170810 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170910 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170810 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602011037840 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602011037840 Country of ref document: DE Representative=s name: MUELLER HOFFMANN & PARTNER PATENTANWAELTE MBB, DE Ref country code: DE Ref legal event code: R081 Ref document number: 602011037840 Country of ref document: DE Owner name: FORTEM SOFTWARE INC., MONTREAL, CA Free format text: FORMER OWNER: FORTEM SOLUTIONS INC., MONTREAL, QUEBEC, CA |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 8 |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: 732E Free format text: REGISTERED BETWEEN 20180315 AND 20180326 |
|
26N | No opposition filed |
Effective date: 20180213 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: TP Owner name: FORTEM SOFTWARE INC., CA Effective date: 20180725 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20180331 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180325 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180325 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180331 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180331 Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180331 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180325 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20110325 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170510 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170510 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20210323 Year of fee payment: 11 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20210319 Year of fee payment: 11 Ref country code: GB Payment date: 20210308 Year of fee payment: 11 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R119 Ref document number: 602011037840 Country of ref document: DE |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20220325 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220325 Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220331 Ref country code: DE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20221001 |