GB2567150A - Method and device for optimizing the search for samples at a video management system - Google Patents

Method and device for optimizing the search for samples at a video management system Download PDF

Info

Publication number
GB2567150A
GB2567150A GB1715836.1A GB201715836A GB2567150A GB 2567150 A GB2567150 A GB 2567150A GB 201715836 A GB201715836 A GB 201715836A GB 2567150 A GB2567150 A GB 2567150A
Authority
GB
United Kingdom
Prior art keywords
media data
source device
method
plurality
obtained
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
GB1715836.1A
Other versions
GB201715836D0 (en
Inventor
Bellessort Romain
Tocze Lionel
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Priority to GB1715836.1A priority Critical patent/GB2567150A/en
Publication of GB201715836D0 publication Critical patent/GB201715836D0/en
Publication of GB2567150A publication Critical patent/GB2567150A/en
Application status is Pending legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/7837Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • G06F16/48Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/5866Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/68Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/686Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/7867Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings

Abstract

A request is received (200) for searching media (e.g. video or audio) data for a specific content (e.g. an object such as car, license plate, person, associated with a context such as weather, day/night, time), the data being obtained by at least one of a plurality of source devices configured to obtain media data. A subset comprising at least one device from the plurality is selected (230), each device being assigned a category based on the request and on an analysis of its obtained data (e.g. sampled), the selecting being based on the assigned category. The media data obtained by the at least one device of the selected subset is searched (240) for the specific content, thereby optimizing the search for samples. The most relevant sources (e.g. cameras) may therefore be searched, with those sources having little relevance to the object or context being avoided.

Description

METHOD AND DEVICE FOR OPTIMIZING THE SEARCH FOR SAMPLES AT A VIDEO MANAGEMENT SYSTEM

FIELD OF THE INVENTION

The present invention relates generally to video surveillance networks.

More specifically, the present invention provides a method and corresponding device for optimizing the search for samples of media data at a video management system, a video management system, and a server-client system comprising a plurality of video management systems.

BACKGROUND OF THE INVENTION

The present invention deals with a Video Management System or Software (VMS) which manages Video Source Devices (VSD) that can stream video data.

Typically, a network camera belonging to a video surveillance system is a video source device as it can capture video and stream it. As another example, a device that obtains a video stream from another device (e.g. a network camera) and that can stream it may be considered as a video source device. This may be the case of a server that obtains streams from a set of network cameras and that merge them to generate an output stream.

Generally speaking, a VMS can discover VSD, get their streamed data and store these data. It also enables a user to display the obtained data (both live streams and previously captured streams). A VMS may have various additional features. For instance, it may enable the settings of the cameras to which it is connected to be changed. It may also enable it to be defined that Video Content Analytics (VCA) should be run on certain streams, and allow a user to search for a specific video stream, e.g. based on time. For instance, a VCA may be dedicated to the determination of the presence of various objects such as vehicles, human beings or animals, or to recognition tasks such as license plate recognition or face recognition. Other sample usages of VCA include intrusion detection, abandoned objects detection or object counting (e.g. cars or people counting).

In most cases, a single VMS is used at a given site (e.g. a company facility). However, if a company has various sites, one VMS may be installed on each site (multi-VMS architecture).

In the context of video surveillance, the VMS manages large amounts of video samples. Samples can be searched for based on location, time, optical properties of the camera, or possibly based on the results of Video Content Analysis tools (VCAs). However, running VCAs on video streams has a significant computing cost, which requires expensive servers.

Patents documents US 8,542,276 or US 8,174,572 propose to analyze recordings to determine relationships between cameras for performing object tracking.

However, there is still a need for means enabling efficient searches to be performed while keeping the processing cost limited.

SUMMARY OF THE INVENTION

The present invention has been devised to address one or more of the foregoing concerns.

According to a first aspect of the invention, there is provided a method for optimizing the search for samples of media data at a video management system configured for managing a plurality of source devices configured to obtain media data, the method comprising the following steps: receiving a request for searching media data obtained by at least one source device of the plurality for a specific content; selecting a subset comprising at least one source device from among the plurality of source devices, each source device of the plurality being assigned a category based on the request and on available results of an analysis of its obtained media data, the selecting being based on the assigned category; searching the media data obtained by the at least one source device of the selected subset for the specific content, thereby optimizing the search for samples at the video management system.

Therefore, an efficient search may be performed at a reduced computing cost.

This is thanks to the categorization of the source devices which allows a subset of source devices to be selected.

Thus, the search is much more optimized than if the media data obtained by all the source devices were considered as in the prior art.

In addition, the present invention is applicable to both single-VMS and multi-VMS architectures.

Optional features of the invention are further defined in the dependent appended claims.

According to embodiments, selecting a subset comprising at least one source device comprises determining a target category based on the request.

According to embodiments, selecting a subset further comprises selecting each source device of the plurality of which the assigned category matches the target category.

According to embodiments, selecting a subset further comprises: for each source device of the plurality, computing a score associated with the assigned category, selecting each source device of the plurality of which the assigned category matches the target category and of which the score belongs to a predefined range.

According to embodiments, the specific content to be searched for defines an object associated with a context in which the object has been captured.

According to embodiments, the context defines at least one of the following elements: weather conditions, whether it is day or night, brightness, a specific day of the week, a specific time of the day, a specific month of the year, the occurrence of a specific event, a specific location.

According to embodiments, the object is at least one of the following elements: a car, a license plate, a person.

According to embodiments, the method comprises a step of assigning a category to each source device of the plurality including identifying, based on the request, the context associated with the specific content to be searched for, wherein assigning further comprises, for each source device of the plurality: obtaining available results from an analysis of its obtained media data; determining, based on the obtained results, whether the source device has already captured the object to be searched for in the identified context; and if so, assigning the target category to the source device; otherwise, determining whether at least part of the obtained results are associated with the identified context; and if so, assigning a category other than the target category to the source device; otherwise, assigning the target category to the source device.

According to embodiments, determining whether the source device is configured to capture the object to be searched for in the context may be based on a detection range.

According to embodiments, the method comprises a step of analyzing obtained media data including: in order to obtain analytics results identifying a set of objects, running analytics on a stream of media data obtained for each context of a set of contexts; storing analytics results in association with the corresponding context.

According to embodiments, the step of analyzing obtained media data is performed according to a sampling policy defining at what time the stream of media data is analyzed.

According to embodiments, the method further comprises, upon detection of an update ofthe video management system, updating the sampling policy.

According to embodiments, the sampling policy defines a periodicity.

According to embodiments, the sampling policy defines different intervals of time during which the stream is more or less regularly analyzed.

According to embodiments, the sampling policy is based on the considered context and/or on the current workload ofthe video management system.

According to embodiments, the sampling policy defines an order in which the media data obtained by the different source devices are analyzed, the order being based on the date of connection of the source devices to the video management system.

According to embodiments, the media data comprise video data and/or audio data.

According to a second aspect of the invention, there is provided a device for optimizing the search for samples of media data at a video management system configured for managing a plurality of source devices configured to obtain media data, the device being configured to perform the following steps: receiving a request for searching media data obtained by at least one source device of the plurality for a specific content; selecting a subset comprising at least one source device from among the plurality of source devices, each source device of the plurality being assigned a category based on the request and on available results of an analysis of its obtained media data, the selecting being based on the assigned category; searching the media data obtained by the at least one source device of the selected subset for the specific content, thereby optimizing the search for samples at the video management system.

According to a third aspect of the invention, there is provided a video management system comprising a device as aforementioned.

According to a fourth aspect of the invention, there is provided a server-client system comprising a plurality of video management systems and a search client implementing a method as aforementioned.

The second, third and fourth aspects of the present invention have features and advantages similar to the first above-mentioned aspect.

Since the present invention may be implemented in software, the present invention may be embodied as computer readable code for provision to a programmable apparatus on any suitable carrier medium, and in particular a suitable tangible carrier medium or suitable transient carrier medium. A tangible carrier medium may comprise a storage medium such as a floppy disk, a CD-ROM, a hard disk drive, a magnetic tape device or a solid state memory device or the like. A transient carrier medium may include a signal such as an electrical signal, an electronic signal, an optical signal, an acoustic signal, a magnetic signal or an electromagnetic signal, e.g. a microwave or RF signal.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described, by way of example only, and with reference to the following drawings in which:

Figure 1a illustrates an exemplary architecture of a VMS system according to embodiments;

Figure 1b illustrates an exemplary architecture of a server-client system comprising a plurality of VMS according to embodiments;

Figure 2 illustrates steps of a method for optimizing the search for samples according to first embodiments;

Figure 3 illustrates steps for assigning a category to VSD according to embodiments;

Figure 4 illustrates steps for analyzing obtained media data according to embodiments;

Figure 5 illustrates steps of a method for optimizing the search for samples at a video management system according to second embodiments.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

The present invention relates notably to a method for optimizing the search for samples at a video management system configured for managing a plurality of source devices each configured to obtain media data (i.e. video data with or without audio, or audio data). A source device may obtain media data in two ways: - either directly if the source device comprises a camera and can thus capture media data directly, or - indirectly if the source device does not comprise a camera but is connected to one or more camera(s) from which the source device can retrieve media data.

According to embodiments, a request for searching media data obtained by at least one source device of the plurality for a specific content is received.

Typically, the specific content to be searched for defines an object, optionally associated with a context in which the object has been captured. For instance, the object may be a car, a car of a certain size, a car of a certain color, a license plate, a person, a person with a given size, etc. The context may define the weather conditions (e.g. sunny weather, rainy weather, foggy weather), whether it is day or night, brightness, a specific day of the week, a specific time of the day, a specific month of the year, the occurrence of a specific event, a specific location.

Next, a subset comprising at least one source device is selected from among the plurality of source devices, based on a category assigned to each source device of the plurality, based on the request, and based on the available results of an analysis of its obtained media data. Finally, the media data obtained by the at least one source device of the selected subset is searched for the specific content.

In this way, the search for samples at the video management system is optimized.

In the following description, the terms “to obtain media data by the source device” may refer to the media data of a live stream or of a previously captured stream. It should be noted that the storage of media data may be done at the source device itself or at another device connected to the source device, such as the VMS (see below).

In the following description, the term “sample” refers to the media data and associated metadata. Such metadata may comprise indications about the video stream itself such as capture conditions and coding parameters, but also about other video streams related to the considered video stream.

Figure 1a illustrates an exemplary architecture of a VMS system according to embodiments.

In this example, the VMS system is configured to manage a plurality of VSD 100. Through a service 105, the VMS obtains and stores media data (typically video data and possibly audio data) as well as associated metadata obtained from the VSD 100, in a dedicated database 110.

The VMS system also comprises a search client 115 which is configured to allow a user to search for specific samples. The search client typically interacts with the database 110.

According to embodiments, the VMS system also comprises an additional service 120 which is configured to perform a categorization of the VSD 100. This categorization service is configured to interact with the VSD 100 and/or the database 110 in order to determine what kind of content (object, context) may or may not be present in the media data obtained by each VSD.

Based on the results of this processing, the categorization service 120 stores metadata which indicate whether a given VSD belongs to a given category.

The present invention is not limited to this example. In other embodiments, different components/modules may be used. For instance, the result of categorization may be stored in a dedicated database distinct from the database 110 used to store media data and metadata. Other components may also be used that are not represented on this figure, such as a configuration service or an administration service.

In practice, the VMS system may be distributed over multiple devices. For instance, a dedicated server may be used to run the service 105 and the database 110.

Figure 1b illustrates an exemplary architecture of a server-client system comprising a plurality of VMS according to embodiments.

In this example, a set of VMS 140, 141 and 142 is considered. Each VMS is made accessible through a VMS Gateway (respectively 150, 151 and 152). The gateway may be a part of the considered VMS which enables accessing the VMS features. Alternatively, the gateway may be implemented as a service running on the Cloud or at a specific remote site from which all VMS are interconnected. A search client 165 enables searching streams obtained by the VSD of each VMS for a specific content (object, context).

According to embodiments, a categorization service 160 is also provided. This service has the same role as the component 120 of Figure 1a. However, a difference between them is that the categorization service 160 can interact with the VSD of various VMS systems, whereas the categorization service 120 interacts with the VSD of a single VMS system.

Therefore, the categorization service 120 can be implemented as a local service by the VMS system of Figure 1a, whereas the categorization service 160 is more likely to be implemented as e.g. a Cloud service. However, a Cloud service could also be used in the context of Figure 1a, and local services could be used in the context of Figure 1b.

When searching the obtained streams for a specific content, the search client 165 can take advantage of the results produced by the categorization service 160. These results may be stored at different places. If stored by the categorization service 160, then the search client 165 interacts with the categorization service 160 to determine which VSD may be relevant for a given search inside each VMS system.

Alternatively, the search client 165 may store the results, or each VMS (possibly each VMS gateway) may store them. In this last case, the search client 165 may either determine the relevant VSD for a given search through each gateway, or it may simply send a search query to each gateway, which would then be responsible for taking advantage of the categorization results.

In the case of a multi-VMS architecture, the time may not be precisely synchronized between each VMS and the categorization service 160, the search client 165 or any device interacting with the VMS. Therefore, when previous samples are obtained from a VMS, the categorization service 160 (or the search client 165 or any device interacting with the VMS) should determine whether there is a shift between its own time and that of a considered VMS. For illustration purpose, if there is a +1 minute time shift (i.e. the considered VMS is 1 minute ahead), the categorization service 160 knows that instead of requesting samples captured at 10:00 am, it should request samples captured at 10:01 am.

Figure 2 illustrates steps of a method for optimizing the search for samples according to first embodiments.

First, at step 200, a request for searching a specific content in media data is received. For illustration purposes, a request may target all the VSD that have obtained media data showing red cars between 4 am and 6 am on a given day, or all the VSD that have obtained media data showing people with a sufficient quality to enable their identification between 7 pm and 8 pm on another given day.

At step 210, a target category is determined based on the request. For illustration purposes, if the request is about “a red car between 4 am and 6 am on a given day” and that VSD is a camera, the target category may be “a camera that captures car”, “a camera that captures car before 8 am”, “a camera that captures car at night”, etc.

Based on the considered day, additional information may be taken into account. For instance, if it is determined that on the considered day, between 4 am and 6 am, there was a lot of fog in the region where VSD are located, the target category may be “cameras that may capture cars even on foggy days”.

As another example, if the request is to search for a car with a specific license plate, the category may be “cameras that may capture license plate even on foggy days”.

Therefore, as it can be seen from these examples, a category may have greater or lesser complexity, and it may involve contextual elements that may influence detection of the targeted object and its usage by analysis tools, e.g. based on spatiotemporal information (e.g. weather, brightness, specific events that occurred at corresponding time in a given location).

Hence, a category may be defined as an object that can be detected in a video stream (e.g. a car, a car of a given size, a car of a given colour, a license plate, a license plate with specific characters, a person, a man wearing specific clothes, a woman with a specific haircut, a face that can be used to identify a person, etc.) and optionally some contextual elements (e.g. night or day, weather conditions, etc.).

At step 220, an actual category is determined for each VSD based on the result of an analysis of its obtained media data. In embodiments, each VSD is assigned with the target category or a “non-target” category. This step is further described with reference to Figure 3.

At step 230, the subset of VSD whose actual category matches the target category is returned.

Optionally, the selection of the subset of VSD may comprise computing a score associated with the actual category for each source device of the plurality so that only the VSD having a score belonging to a predefined range and whose actual category matches the target category are returned.

For instance, this score may be based on the number of times an object appears in media data obtained by a considered VSD, or on a frequency of detection (e.g. 1 car I minute at night). The VSD for which no detection of a given object in a given context has been observed may be assigned a score of 0. For instance the predefined range may define a minimum and a maximum number (or frequency) of detections. In practice, this predefined range may be provided in the search request.

This optional score computing is advantageous since a lower number of VSD is finally selected and thus the search is further simplified.

At step 240, the media data obtained by the video source devices of the subset is searched for the specific content indicated in the search request received at step 210.

Finally, the process ends at step 290.

Figure 3 illustrates steps for assigning a category to a VSD according to embodiments. These steps are preferably performed for each VSD.

At step 310, the results of the analysis of the VSD media data are retrieved. The process enabling these results to be obtained is described with reference to Figure 4.

At step 320, it is determined, based on the results retrieved at step 310, whether the VSD has already obtained media data comprising the object targeted in the requested specific content in the considered context, or alternately, whether the VSD has obtained media data comprising this object in the considered context with a sufficient detection score.

For illustration purposes, it may consist of determining whether a VSD has already obtained media data in which license plates were detected when the weather was rainy.

In the case where a detection score is used, the step 320 is based on a detection range. For instance, if the target object is a person, it may be decided that at least 1 person per day should be detected by a given video source device in order for the test of step 320 to be true (i.e. the detection score is sufficient).

In practice, this predefined range may be provided in the search request. Such feature may be useful when there is a large number of samples to be processed as it may allow some types of VSD to be selected based on the context of the search (e.g. first start by searching only VSD that obtain media data with a large number of target objects, and if no satisfying result is found, focus on VSD that obtain media data with a lower number of target objects).

When the determination is positive (object already detected with a sufficient detection score in the considered context), the VSD is assigned “TC” as actual category at step 330.

When the determination is negative (object never detected or with an insufficient detection score in the considered context), it is determined at step 340 whether some of the obtained results are associated with a context similar (identical) to the one defined in the request (e.g. by TC contextual elements). If there are no such contextual elements, which may happen for instance if the requested specific content is simply characterized by a target object such as license plate, then the test 340 is always considered to be true.

If none of the obtained results are associated with a similar context, the VSD should also be assigned "TC" as actual category (step 330). This is because, as a matter of fact, this does not exclude that the VSD may obtain media data where such objects appear in the considered context in the future.

On the other hand, in case of positive answer (at least some of the obtained results are associated with a similar context), the step 340 is followed by the step 350, which consists in assigned “not TC” as actual category for VSD.

Steps 330 and 350 are followed by the end of the process for this VSD (step 390).

Figure 4 illustrates steps for analyzing obtained media data according to embodiments. These steps may be performed as a preliminary phase. They are preferably performed for each VSD.

At step 410, a captured stream is obtained by the VSD, ideally for each context of a set of contexts.

In practice, this may be done in two ways: - First, it may be possible to rely on previously captured video streams (as well as potential audio stream and/or metadata stream) that may be stored by the VMS associated with the VSD. The advantage of relying on such data is that it is readily available to start the categorization process; - Second, the live video stream (as well as potential audio stream and/or metadata stream) may be obtained by a given VSD at given times based on the contexts. For instance, a sampling policy may be defined in order to determine when media data may or may not be obtained by a given VSD. For illustration purposes, 30 seconds of live video stream may be obtained every hour for 1 week. In embodiments, the sampling policy also takes into account the fact that multiple VSD will typically be sampled. Therefore, sampling different VSD at the same time should be avoided in order to minimize the amount of resources necessary to perform this process.

When some streams retrieved at step 410 are streams that have been previously captured (as opposed to live video streams), and if the VMS runs VCA that are known to be reliable (see description of Figure 5), it may not be needed to run a VCA at step 420. In this case, the results of VCA that have already been run by the VMS for the considered streams may be directly used.

At step 420, VCA algorithms are run on the streams obtained at step 410. Typically, there may be one VCA algorithm per considered kind of object (license plate, person, etc.). It may happen that no VCA is defined for a given kind of object. In this case, it cannot be determined whether said kind of object may be captured or not in the media data obtained by the VSD.

The results of step 420 are stored at step 430. For instance, it is stored that in a given context, two occurrences of a given object were detected in a given amount of time. Alternatively, it may simply be stored that occurrences of a given object were detected in the media data obtained by the VSD. If no distinction is made between different contexts, it is assumed that only a single generic context is considered, and the result of the process may simply be a Boolean value.

Storing a numerical value may allow finer grain processing, e.g. by defining a threshold for considering a VSD as belonging to the target category, or to enable ranking of the video source devices according to a score.

In the case where a VCA algorithm returns some data associated to the media data it has processed, said data may also be stored. For instance, a given region of interest in the processed video may be stored as metadata. A level of confidence regarding the detection may also be indicated by the VCA and stored as a result.

In embodiments, a VSD may be associated with some metadata indicating that the VSD cannot detect a given object (e.g. because its resolution is very limited).

In such cases, steps 410 and 420 could be skipped, so that an indication that the VSD is not adapted to detect a considered object may be directly stored at step 430.

However, performing steps 410 and 420 anyway may be useful since metadata may not be reliable and the computing cost of these steps is relatively limited. Advantageously, by running steps 410 and 420 even for such VSD, the process reduces its dependency on metadata reliability.

Generally speaking, the sampling policy defines at what time the stream obtained by a VSD should be analyzed.

According to embodiments, the sampling policy may merely define a periodicity.

According to embodiments, the sampling policy may define different intervals of time during which the stream is more or less regularly analyzed. For example, specific time intervals may be defined (e.g. specific time ofthe day or specific day of the week) that should be more sampled than others. This may be visible in the set of contexts considered (e.g. each specific time interval may be a context).

According to embodiments, the sampling policy may be based on the considered context. Thus, if the context involves external factors such as weather condition, the sampling policy may take such factors into account. This may be done by connecting the service responsible for determining a sampling policy to a service indicating the weather condition of the considered place. For instance, if some data are missing for rainy weather and that the weather appears to be rainy at a given time, then it may be decided to sample streams at this time.

According to embodiments, the sampling may advantageously take into consideration the current workload of the different components involved. For instance, getting multiple streams from the same VMS at the same time should be avoided. As another example, if the server used to run VCA at step 420 is used by another process (e.g. to identify a specific person that is currently being searched in the context of an important investigation), the sampling (or at least the analysis ofthe stream) should be postponed to another time when the server does not have to process a higher priority task.

According to embodiments, the sampling policy may define an order in which the media data obtained by the different VSD are analyzed, the order being based on the date of connection of the source devices to the video management system. This is because VMS system(s) can be modified over time. For instance, one VSD may be moved from one place to another, some may be removed, others added.

Therefore, it is advantageous to determine new VSD (e.g. based on their ID, which was previously unknown), VSD that have been moved (e.g. based on their location), and VSD that have been removed (e.g. ID is no longer present in the list of IDs for registered video source devices) so as to apply a different process to them.

In particular, new VSD and VSD that have been moved should be sampled in priority in order to quickly determine whether they may detect potential target objects in the considered contexts.

As described above, if any samples have been stored for such VSD (for a moved VSD, only samples associated with the new location should be considered), these samples may be relied on. In addition, the sampling frequency may be increased for such VSD. Conversely, for VSD located at a given place for a long time, the sampling frequency may be decreased.

As a remark, it should be noted that even if the sampling frequency may be decreased, sampling should preferably not be stopped. As a matter of fact, by going on with the sampling, results consistency can be checked. It may in particular be detected that the context and/or object of the media data obtained by a given VSD has changed. This may happen if the video source device has been moved or if the place that is captured has changed. For instance, if there are works on a road, it may happen that no vehicle goes through that road for several weeks. As a consequence, it is also advantageous to enable storage of different results (at step 430) for different periods of time. This may be part of the context.

Figure 5 illustrates steps of a method for optimizing the search of samples for specific content at a video management system according to second embodiments.

Such algorithm may be implemented in the search client of the VMS (Figure 1a), or in a dedicated search client that may interact with a set of VMS (Figure 1b). It should be noted that in the case of Figure 1b, this algorithm may also be implemented by each gateway or VMS. In this case, the gateway receives a generic search request, and it applies the algorithm (or it forwards the request to a VMS component which applies the algorithm).

First, at step 500, a request for searching an object (e.g. a car or a person) in a given area (e.g. a part of a city) and/or for a given period of time (e.g. between 10 am and 11 am on a given day) is received.

It is assumed that the object is associated with a video content analytics that allows determination of whether it is comprised in a sample of media data.

At step 510, the relevant VSD based on the considered area and period of time are determined. This may be done based on the location of each VSD that is reachable by the search system. Alternatively, it may simply be done based on the location of the VMS that handles each VSD.

At step 520, it is determined whether there remains a VSD which has not yet been processed.

If so, it is determined at step 530 whether a specific VCA enabling detection of the target object is running on the VSD stream.

If so, it is determined at step 540 whether this VCA is reliable.

The reliability of a VCA can be determined by comparing its results to those of another similar VCA used as a reference. For illustration purposes, two VCA (possibly the same software but with different configurations) may be able to detect cars in an image, but they may not always provide the same results. For instance, a VCA may detect a car even though there is no car, or it may not detect a car even though there is one.

This situation may in particular occur with a multi-VMS architecture. For illustration purposes, a search client may be hosted by a given organization (e.g. a Law Enforcement Agency) that is connected to several remote VMS. Each of these VMS may run its own VCA (to detect cars), but the organization may also have its own VCA. If the search client implements the invention, the VCA of the remote VMS should be relied on only if it provides results similar to the ones obtained using the organization’s own VCA.

If this VCA is reliable (positive answer to test 540), the media data obtained by the VSD can be submitted to a VCA search using this reliable VCA. This is because since the VCA is reliable and is run on the stream obtained by the VSD, the relevant samples for this VSD can be obtained simply by requesting VSD samples associated with a positive detection for the considered VCA. The results of the VCA search are obtained at step 550.

On the other hand, if the considered VCA is not handled by the VSD (negative answer to test 530) or if it is not reliable (negative answer to test 540), the VSD is added to the set of devices to be filtered (step 560). This allows the number of considered VSD to be minimized, hence the number of samples. The step 560 (and step 550) is followed by the step 520.

When there is no remaining VSD (negative answer to test 520), the set of VSD to be filtered is filtered at step 570 according to steps 210 to 230 of Figure 2.

At step 580, the media data obtained by the filtered VSD is searched for the specific content indicated in the search request received at step 500.

The process then ends at step 590.

As a remark, the filtering of step 570 may be applied to all relevant VSD determined at step 510 before running step 520. In this case, when the answer to test 520 is negative, this step is directly followed by the step 580, which is applied to the set of VSD built at step 560. This embodiment may be useful if categorization information is available even for VSD for which VCA are always executed. Indeed, this allows requesting such VMS at step 550 only for VSD that are likely to provide results.

An example of application of this process is now described for illustration purposes only.

In this example, a user would like to search for a car (object) with a given license plate which has been seen in a given area of Paris between 10 am and 11 am on a given rainy day (context).

The user inputs a query with these parameters at a search client. The search client determines that one VMS has cameras located in the considered area. The search client also determines that this VMS cannot run a VCA that can detect license plates.

However, the search client has previously sampled the cameras of this VMS and determined that among its 50 cameras, 30 of them never capture any license plate (e.g. because they do not capture vehicles due to their location, or because their optical properties do not allow a sufficient resolution for reading license plates), and that 10 of them capture license plates, but that image quality is not sufficient to read them in case of rain. Therefore, the search client knows that only 10 cameras may capture readable license plates when the weather is rainy.

Therefore, the search client sends a request to the VMS in order to obtain the samples for those 10 cameras at the considered time. The samples are returned. The user may then be able to process the samples using a dedicated license plate video analytics that may find the relevant license plate. Consequently, thanks to the categorization of cameras, the number of samples to be considered can be greatly reduced, hence the result much more quickly determined.

While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive, the invention being not restricted to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in putting into practice (i.e. performing) the claimed invention, from a study of the drawings, the disclosure and the appended claims.

In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfil the functions of several items recited in the claims. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be advantageously used. Any reference signs in the claims should not be construed as limiting the scope of the invention.

Claims (20)

1. A method for optimizing the search for samples of media data at a video management system configured for managing a plurality of source devices configured to obtain media data, the method comprising the following steps: receiving a request for searching media data obtained by at least one source device of the plurality for a specific content; selecting a subset comprising at least one source device from among the plurality of source devices, each source device of the plurality being assigned a category based on the request and on available results of an analysis of its obtained media data, the selecting being based on the assigned category; searching the media data obtained by the at least one source device of the selected subset for the specific content, thereby optimizing the search for samples at the video management system.
2. The method of claim 1, wherein selecting a subset comprising at least one source device comprises determining a target category based on the request.
3. The method of claim 2, wherein selecting a subset further comprises selecting each source device of the plurality of which the assigned category matches the target category.
4. The method of claim 2, wherein selecting a subset further comprises: for each source device of the plurality, computing a score associated with the assigned category, selecting each source device of the plurality of which the assigned category matches the target category and of which the score belongs to a predefined range.
5. The method of any one of claims 1 to 4, wherein the specific content to be searched for defines an object associated with a context in which the object has been captured.
6. The method of claim 5, wherein the context defines at least one of the following elements: weather conditions, whether it is day or night, brightness, a specific day of the week, a specific time of the day, a specific month of the year, the occurrence of a specific event, a specific location.
7. The method of claim 5 or 6, wherein the object is at least one of the following elements: a car, a license plate, a person.
8. The method of any one of claims 5 to 7, comprising a step of assigning a category to each source device of the plurality including identifying, based on the request, the context associated with the specific content to be searched for, wherein assigning further comprises, for each source device ofthe plurality: obtaining available results from an analysis of its obtained media data; determining, based on the obtained results, whether the source device has already captured the object to be searched for in the identified context; and if so, assigning the target category to the source device; otherwise, determining whether at least part of the obtained results are associated with the identified context; and if so, assigning a category other than the target category to the source device; otherwise, assigning the target category to the source device.
9. The method of claim 8, wherein determining whether the source device is configured to capture the object to be searched for in the context may be based on a detection range.
10. The method of claim 8 or 9, comprising a step of analyzing obtained media data including: in order to obtain analytics results identifying a set of objects, running analytics on a stream of media data obtained for each context of a set of contexts; storing analytics results in association with the corresponding context.
11. The method of claim 10, wherein the step of analyzing obtained media data is performed according to a sampling policy defining at what time the stream of media data is analyzed.
12. The method of Claim 11, further comprising, upon detection of an update of the video management system, updating the sampling policy.
13. The method of claim 11 or 12, wherein the sampling policy defines a periodicity.
14. The method of claim 11 or 12, wherein the sampling policy defines different intervals of time during which the stream is more or less regularly analyzed.
15. The method of any one of claims 11 to 13, wherein the sampling policy is based on the considered context and/or on the current workload of the video management system.
16. The method of any one of claims 11 to 15, wherein the sampling policy defines an order in which the media data obtained by the different source devices are analyzed, the order being based on the date of connection of the source devices to the video management system.
17. The method of any one of claims 1 to 16, wherein the media data comprise video data and/or audio data.
18. A device for optimizing the search for samples of media data at a video management system configured for managing a plurality of source devices configured to obtain media data, the device being configured to perform the following steps: receiving a request for searching media data obtained by at least one source device of the plurality for a specific content; selecting a subset comprising at least one source device from among the plurality of source devices, each source device of the plurality being assigned a category based on the request and on available results of an analysis of its obtained media data, the selecting being based on the assigned category; searching the media data obtained by the at least one source device of the selected subset for the specific content, thereby optimizing the search for samples at the video management system.
19. A video management system comprising a device according to claim 18.
20. A server-client system comprising a plurality of video management systems and a search client implementing a method according to claim 1.
GB1715836.1A 2017-09-29 2017-09-29 Method and device for optimizing the search for samples at a video management system Pending GB2567150A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
GB1715836.1A GB2567150A (en) 2017-09-29 2017-09-29 Method and device for optimizing the search for samples at a video management system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB1715836.1A GB2567150A (en) 2017-09-29 2017-09-29 Method and device for optimizing the search for samples at a video management system
PCT/EP2018/075475 WO2019063409A1 (en) 2017-09-29 2018-09-20 Method and device for optimizing the search for samples at a video management system

Publications (2)

Publication Number Publication Date
GB201715836D0 GB201715836D0 (en) 2017-11-15
GB2567150A true GB2567150A (en) 2019-04-10

Family

ID=60270486

Family Applications (1)

Application Number Title Priority Date Filing Date
GB1715836.1A Pending GB2567150A (en) 2017-09-29 2017-09-29 Method and device for optimizing the search for samples at a video management system

Country Status (2)

Country Link
GB (1) GB2567150A (en)
WO (1) WO2019063409A1 (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080080745A1 (en) * 2005-05-09 2008-04-03 Vincent Vanhoucke Computer-Implemented Method for Performing Similarity Searches

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10275656B2 (en) * 2014-11-21 2019-04-30 Trustees Of Boston University Large scale video search using queries that define relationships between objects
WO2017046704A1 (en) * 2015-09-14 2017-03-23 Logitech Europe S.A. User interface for video summaries

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080080745A1 (en) * 2005-05-09 2008-04-03 Vincent Vanhoucke Computer-Implemented Method for Performing Similarity Searches

Also Published As

Publication number Publication date
GB201715836D0 (en) 2017-11-15
WO2019063409A1 (en) 2019-04-04

Similar Documents

Publication Publication Date Title
KR101810578B1 (en) Automatic media sharing via shutter click
CA2722300C (en) Method and system for analyzing multimedia content
US20080273088A1 (en) Intelligent surveillance system and method for integrated event based surveillance
US7760230B2 (en) Method for automatically reducing stored data in a surveillance system
US20130166711A1 (en) Cloud-Based Video Surveillance Management System
US20090271529A1 (en) Stream data processing method and computer systems
US20080109369A1 (en) Content Management System
CN103004188B (en) Devices, systems and methods
US20140043480A1 (en) Video monitoring system and method
US9230172B2 (en) Systems and methods for image-feature-based recognition
US20190014289A1 (en) Systems and methods for automated cloud-based analytics for security and/or surveillance
US20090077049A1 (en) Restoring Program information for Clips of Broadcast Programs Shared Online
AU2008262268A1 (en) System and method for integrating video analytics and data analytics/mining
JP2004166024A (en) Monitoring camera system and monitoring method
CN103380619B (en) Video Search record
CN101918989A (en) Video surveillance system with object tracking and retrieval
CN101281593B (en) Method and system for researching intelligent video monitoring case
US20070058842A1 (en) Storage of video analysis data for real-time alerting and forensic analysis
US9560323B2 (en) Method and system for metadata extraction from master-slave cameras tracking system
US9122928B2 (en) Determining images having unidentifiable license plates
US9710924B2 (en) Field of view determiner
US9141184B2 (en) Person detection system
US9208226B2 (en) Apparatus and method for generating evidence video
US20140369566A1 (en) Perimeter Image Capture and Recognition System
US20130054598A1 (en) Entity resolution based on relationships to a common entity