DE102021207142A1 - Surveillance arrangement, surveillance method, computer program and storage medium - Google Patents

Abstract

Monitoring arrangement 1 with a central module 3 and at least one client module 2, image data 4 being provided to client module 2, client module 2 having a processing unit 6, an evaluation unit 5 and a plurality of evaluation applications 8a,b, evaluation unit 5 being designed to Execute and/or use evaluation applications 8a,b, wherein the evaluation applications 8a,b are each designed to detect areas of interest in the image data 4 when executed and/or used, the processing unit 6 being designed based on the detected areas of interest of the respective evaluation application 8a, b and the image data 4 to generate application-specific image data 8a,b, the application-specific image data 8a,b comprising and/or describing image content of the areas of interest with a minimum fidelity, the client module 2 having a provision unit 15, the provision unit 15 is designed to provide the application-specific image data 8a,b to the central module 3, with the central module 3 having a combination unit 12, with the combination unit 12 being designed to merge the application-specific image data 8a,b into a merged image, with the merged image combining the image contents of the areas of interest in a minimum fidelity.

Description

State of the art

The invention relates to a surveillance arrangement with a central module and at least one client module, image data being provided to the client module.

Camera-based surveillance is used in the public, private and commercial sectors. The images and/or videos recorded by the cameras are often sent to a control center for storage and/or evaluation. The quality of the recorded images and/or videos is often very good, for example 4K, so that their transmission to the control center means a large data stream during transmission.

For example, the reference describes US20140279982A1 a method and a device for managing a plurality of images, wherein a plurality of images of different qualities are generated for storage.

Disclosure of Invention

The invention relates to a monitoring device having the features of claim 1. The invention also relates to a method for monitoring, a computer program and a storage medium. Preferred and/or advantageous configurations result from the dependent claims, the description and the attached figures.

The invention relates to a monitoring arrangement. The monitoring arrangement comprises at least one central module and at least one client module, preferably a plurality of client modules, for example more than ten. The central module can be designed, for example, as a computer, calculator, server or as a cloud. The client modules are preferably arranged in a decentralized manner, for example within a monitored area and/or at a customer. The monitoring arrangement is preferably designed to monitor the monitored area. For example, the client modules are embodied as cameras or sensors for recording and/or monitoring the monitored area and/or are integrated therein, with the central module being embodied, for example, for central viewing, evaluation and/or storage of this data.

Image data is provided to the client module. The image data can form and/or include images, individual images, an image stream or videos. For example, the client modules are connected to a camera in terms of data technology or are part of a camera. The image data is preferably of high quality, for example 4K or HD. The client module has an evaluation unit and a processing unit. In particular, the processing unit and evaluation unit form a common unit. The evaluation unit and/or the processing unit are designed, for example, as software units, alternatively as hardware units, for example comprising a processor or a data processing unit. In particular, the evaluation unit is provided with the image data, in particular unprocessed, uncompressed and/or in full quality.

The monitoring arrangement includes a number of evaluation applications. The evaluation applications are designed as software applications, for example. In particular, the evaluation applications of the monitoring arrangement can be exchanged, expanded and/or can be obtained from a third party, for example an app store. The evaluation applications are stored and/or installed on the evaluation unit, for example. The evaluation unit is designed to execute and/or use the evaluation applications, in particular to execute and/or use them in parallel. In particular, the evaluation applications are independent of one another. In particular, the evaluation unit is designed to apply the evaluation applications to the image data.

When used and/or executed by the evaluation unit, the evaluation applications are designed to detect and/or determine areas of interest in the image data, in particular in the images, individual images, image streams and/or videos comprised therein. In particular, when executing and/or using the evaluation applications, object recognition, image evaluation, object tracking is applied to the image data, preferably based on a neural network. The evaluation applications are specifically designed to determine areas of interest for different evaluations, applications and/or objects. For example, an evaluation application is designed for face recognition, with image sections of the image data with faces being detected as areas of interest, with another evaluation application being designed for full-body person recognition, with image sections showing a person being detected as an area of interest. Areas of interest are, for example, sections and/or areas of the image data, in particular in images and/or videos, which are searched for by a user and/or are relevant for a task or further evaluation and/or are of particular interest. For example, the entire image and/or video is rarely of interest to a user and/or application, for example when the image includes a particularly large number of objects or background. at For example, a static image area or background is uninteresting for the application and/or the user or is of lesser interest than certain objects, for example people, suitcases or weapons. In particular, the evaluation unit is designed to determine and/or provide form, position and/or position as additional data for the detected areas of interest.

The processing unit is designed to generate application-specific image data. In particular, the application-specific image data are determined for all evaluation applications that are executed and/or used. The application-specific image data include, describe and/or show the image content contained therein with a minimum fidelity for the associated areas of interest. The minimum fidelity is in particular a high fidelity, such as the original fidelity or at least 95%, in particular at least 99%, of the original fidelity. For example, the application-specific image data includes image sections and/or image areas of the areas of interest in original quality, e.g. 4K or HD. The minimum fidelity is, for example, a fidelity that is at least required for a further image evaluation, in particular that object features are recognizable, detectable, classifiable and/or segmentable. In particular, the minimum fidelity is better than the fidelity of an encoded and/or compressed entertainment video, for example better than in an MP4 format. In particular, high fidelity and/or minimum fidelity means an image or video quality that is better than that of an image data stream. In particular, fidelity can be understood to mean fidelity in relation to a resolution, a color rendering, a brightness value and/or a number of pixel errors. For example, the image data is transmitted to the central module as an image data stream, with the image data stream comprising the image data in encoded and/or compressed form. In particular, all image areas, including the actual areas of interest, are included in the image data stream with a low and/or lower fidelity.

The application-specific image data is generated in particular with knowledge and/or with the aid of the detected areas of interest. The compressed image data are designed in particular for transmission and/or data transmission. The compressed image data is specifically encoded image data and/or compressed image data. For example, the image data, in particular the image contents of the areas of interest, are encoded and/or compressed for this purpose. In particular, the application-specific image data includes metadata, the metadata describing, for example, a position, size and/or shape of the areas of interest.

The client module includes a deployment unit. The provision unit is designed in particular for the data connection with the central module, for example for the data connection of the client module with the central module, preferably as a wireless connection or as a mixed connection, comprising wireless and wired connection sections. The provision unit is designed to provide the application-specific image data to the central module. In particular, the provision unit can be designed to transmit further data based on the image data in addition to the application-specific image data.

The central module includes a combination unit. The combination unit is designed in particular as a software unit, alternatively as a hardware unit, for example comprising a data processing device, a processor or computer. The combination unit is designed to determine a merged image based on the application-specific image data. The fusion image shows and/or includes the areas of interest in the high fidelity, with areas of non-interest in particular including and/or shown at lower fidelity. For example, the combination unit evaluates the application-specific image data for this purpose, for example the image content of the areas of interest is processed, created, decoded and/or decompressed in the high fidelity, the image content contained in the high fidelity being fused, merged and/or assembled to form the fusion image. In particular, the fusion image can form a fusion video together with a plurality of fusion images. The fusion image and/or fusion video is designed, for example, to be displayed to a user, for further use and/or evaluation.

The invention is based on the idea of not transmitting high-resolution images and/or videos from surveillance as a whole to a central module with minimum fidelity and/or high fidelity, but decentralized, especially in the camera, to pre-evaluate the images or image data and only use areas of interest in to transmit high quality and / or fidelity to the central module. In particular, a large number of independent analyzes are possible on the client module, since only the areas of interest are to be transmitted in high quality.

One embodiment of the invention provides that the processing unit is designed to determine, generate and/or provide the image data stream based on the image data. The image data stream includes and/or describes the image data in compressed, encoded, quality-reduced, resolution-reduced and/or lower fidelity than the original image data. The image data stream is provided and/or sent by the provision unit to the central module, in particular “live” or buffered. The combination unit is designed to process the image data stream for display and/or evaluation, for example to prepare, extract, decode, decompress and/or determine the image content of the image data stream, for example as stream image data. The stream image data includes the image content of the image data with a lower fidelity than the fidelity of the application-specific image data. The combination unit is designed to generate the merged image based on the image data stream, in particular the stream image data, and the application-specific image data. For this purpose, for example, the areas and/or image sections in the stream image data that belong to areas of interest are replaced by the image content of the application-specific image data, specifically with high fidelity. This configuration is based on the idea of only transmitting the relevant areas of interest with high fidelity and/or minimum fidelity and merging them with the application-specific image data to create the merged image, which also shows the areas of no interest, in order to include the corresponding areas with increased quality .

It is optionally provided that the client module and/or the central module has an evaluation unit. In particular, the combining unit includes the evaluating unit. The evaluation unit is designed to evaluate, evaluate and/or compare the application-specific image data, in particular the image content and/or the areas of interest. The assessment unit determines additional information for the application-specific image data, in particular the image content and/or areas of interest. For example, the additional information is determined in each case for all the evaluation applications used and/or executed and/or for the application-specific image data provided to the central module. The additional information describes and/or forms a quality assessment and/or a relevance assessment. The quality assessment describes, for example, the fidelity to reproduction, quality, resolution and/or depth of detail of the image content of the areas of interest and/or the application-specific image data. The relevance rating describes, for example, the relevance, importance, probability of further use of the area or areas of interest. In particular, the additional information can include different quality ratings and/or relevance ratings for different areas of interest in the respective application-specific image data. The combination unit is designed to determine and/or generate the merged image based on the additional information. For example, the application-specific image data and/or image content of the areas of interest used for the fusion is selected based on the additional information, in particular in the case of redundancy and/or overlapping of areas of interest in different application-specific image data.

It is particularly preferred that the assessment unit is designed, in the event of redundancy, multiple presences and/or overlapping areas of interest, to use and/or use the image content from the application-specific image data for fusion to form the fused image, which has the best quality assessment and/or relevance assessment, in particular for the respective area of interest. For example, one of the evaluation applications is designed for face recognition, with image sections with faces being provided as areas of interest in the application-specific image data, with another evaluation application being designed for person tracking and a person's entire body being detected as an area of interest, with the person's entire body being included in the application-specific Image data is provided with a slight reduction in resolution, so that the face of the person is included in both application-specific image data, but the face from the application-specific image data of face recognition is used for the fusion image.

In particular, the quality rating forms a numerical value, for example a percentage. The quality assessment optionally describes and/or forms a signal-to-noise ratio or ratio between information depth per image section, for example bits per pixel.

In one embodiment of the invention, a relevance map is determined. For example, the relevance map is determined by the evaluation application when it is executed and/or used. Alternatively and/or additionally, the relevance map is determined by the evaluation unit. The relevance map comprises a plurality of pixels and/or image sections, the pixels and/or image sections being assigned to the image data, in particular images of the image data. For example, an evaluation application is implemented To recognize a face and/or to determine a bounding box around the face, with central facial features being of greater relevance for a further and/or later evaluation than outer areas or the box, so that a relevance map is created whose pixels are arranged in the inner area of the face.

Provision is optionally made for a user to be able to select and/or define one or more evaluation applications as selected evaluation applications. The central module is provided with the selection and/or definition of the selected evaluation applications. The combination unit is designed to prioritize the image content, image data and/or areas of interest of the selected evaluation applications over the other, in particular non-selected, evaluation applications for merging and/or generating the merged image.

In particular, the application-specific image data include a reference. The reference relates, for example, to image content, areas of interest and/or image data that have already been transmitted. For example, a face in the image data has already been transferred in the application-specific image data, so that instead of the transfer of the image content in the application-specific image data, only a reference is provided.

An embodiment of the invention provides that the evaluation applications apply and/or use a common screening and/or division of the images and/or image data. For example, the rastering is designed as a division into image sections of the same size and/or uniform, for example a division of the images of the image data into rectangles. In this case, the areas of interest are quantized by the rasterization, so that the areas of interest can only form integral multiples of the rasterization and/or raster elements. This embodiment is based on the consideration that in the case of an overlap of areas of interest and the use of the merger, it can be ensured that the areas of interest can each replace a complete raster element and no loss of quality is to be expected.

The provision unit is optionally designed to monitor data traffic and/or utilization of the data connection between the client module and the central module and to determine a data traffic measure. For example, the data traffic measure describes how heavy the utilization of the data connection is, for example as a percentage. The provision unit is preferably designed to manage and/or control the provision of the application-specific image data and/or their transmission to the central module based on the data traffic measure. For example, the provision unit is designed to transmit only the application-specific image data with the highest quality rating and/or relevance rating when the data connection is heavily utilized and/or not to transmit application-specific image data with a low quality rating and/or relevance rating and/or to hold it back until the utilization is lower.

The client module particularly preferably includes a feedback unit. The feedback unit is designed, in particular for the use of redundancy in the application-specific image data, to manage, control, and regulate the processing unit, so that the application-specific image data is generated in such a way that original redundant image sections are only recorded once, in particular with the associated highest fidelity to the original application-specific image data are provided.

A further subject matter of the invention is a method for monitoring a surveillance area, image data being and/or being provided to a client module, in particular from a surveillance camera. A plurality of evaluation applications are executed and/or used on the client module, with the evaluation applications each detecting areas of interest in the image data. Application-specific image data are generated based on the interest data and the image data, wherein the application-specific image data include the image content of the areas of interest in minimum fidelity. The application-specific image data are provided to the central module, with a fusion image being determined on the central module based on the application-specific image data, the fusion image comprising the image contents of the areas of interest with high fidelity.

A further object of the invention is a computer program, the computer program being designed to execute and/or use the method for monitoring when it is executed.

A further object of the invention is a machine-readable storage medium, the computer program being stored on the storage medium.

• 1 ein erstes Ausführungsbeispiel einer Überwachungsanordnung;
• 2 ein Ausführungsbeispiel einer Überwachungsanordnung mit überwachtem Datenverkehr;
• 3 ein Ausführungsbeispiel einer Überwachungsanordnung mit Feedbackeinheit.

Further advantages, effects and configurations result from the attached figures and their description, showing:

• 1 a first embodiment of a monitoring arrangement;
• 2 an embodiment of a monitoring arrangement with monitored data traffic;
• 3 an embodiment of a monitoring arrangement with feedback unit.

1 shows a monitoring arrangement 1. The monitoring arrangement 1 comprises at least one client module 2 and a central module 3. The client module 2 and the central module 3 are connected to one another in terms of data technology. The central module 3 is arranged, for example, centrally at a manufacturer, security service provider and/or a monitoring center. In particular, the central module 3 forms a cloud. The client module 2 is decentralized, for example arranged in a surveillance area. The surveillance area is video-monitored with video cameras, with image data 4 from the video cameras being made available to the client module 2 . The image data 4 are in particular of high quality, for example HD or 4K.

The client module 2 comprises an evaluation unit 5 and a processing unit 6. The evaluation unit 5 is provided with the image data 4. The transmission of the entire image data 4 to the central module 3 in the high quality would lead to a large amount of data traffic. Frequently, not all areas and/or sections of images comprised by the image data 4 are necessary for further processing and/or evaluation, or not all are of good quality.

The client module 2 is designed to provide the central module 3 with application-specific image data 7a, 7b. The evaluation unit 5 is designed to use evaluation applications 8a, 8b on the image data 4, regions of interest being detected in the image data. The processing unit 6 is designed to generate the application-specific image data 8a, b in such a way that the image contents of the areas of interest are included in the application-specific image data 8a, b with a minimum fidelity, in other words a high fidelity. Furthermore, the processing unit 6 is designed to provide image stream data 9, the image stream data 9 comprising the image contents of the image data 4 in compressed and/or encoded form.

The central module 3 has a stream decoding unit 10 for decoding the image stream data 9 and application image data decoding units 11a, b for decoding the application-specific image data 8a, b. Furthermore, the central module 3 has a combination unit 12 for generating the merged image. The merged image forms a merging and/or fusion of the image content of the image stream data 9 with the image content of the application-specific image data 8a, b, so that the image content of the areas of interest are included in the merged image with high fidelity or minimum fidelity, with the other areas having a lower fidelity .

The central module 3 also has a storage unit 13 for storing application-specific image data 8a, 8b in compressed and/or decompressed form. In particular, application-specific image data that has already been stored can be called up from the storage unit 13, for example by the combination unit 12. The central module also has a display unit 14 for displaying the merged image.

2 shows an exemplary embodiment of a monitoring arrangement 1, the client module 2 having a provisioning unit 15 for providing and/or sending the application-specific image data 8a, b and/or the image data stream 9 to the central module 3. The provisioning unit 15 is designed to determine a data traffic measure that indicates the utilization of the Data connection from client module 2 and central module 3 describes. The provision unit 15 is designed here to control and/or adapt the provision of the application-specific image data 8a, b based on the current data traffic measure, for example by withholding, delaying or discarding application-specific image data 8a, b with a low quality rating and/or relevance rating.

3 shows an exemplary embodiment of a monitoring arrangement 1, wherein the client module 2, in particular the provision unit 15, has a feedback unit 16, the feedback unit 16 forming a feedback of the application-specific image data 8a, b to the processing unit 6, so that the processing unit when generating the application-specific image data 8a, b can provide redundant areas of interest only once in the application-specific image data 8a, b.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US20140279982A1 [0003]

Claims (14)

Monitoring arrangement (1) with a central module (3) and at least one client module (2), image data (4) being provided to the client module (2), the client module (2) having a processing unit (6), an evaluation unit (5) and a plurality of evaluation applications (8a,b), wherein the evaluation unit (5) is designed to execute and/or use the evaluation applications (8a,b), wherein the evaluation applications (8a,b) are each designed to detect areas of interest in the image data (4) during execution and/or use, wherein the processing unit (6) is designed to generate application-specific image data (8a,b) based on the detected areas of interest of the respective evaluation application (8a,b) and the image data (4), the application-specific image data (8a,b) including image content of the areas of interest in include and/or describe a minimum fidelity wherein the client module (2) has a provision unit (15), wherein the provision unit (15) is designed to provide the application-specific image data (8a, b) to the central module (3), wherein the central module (3) has a combining unit (12), wherein the combining unit (12) is designed to merge the application-specific image data (8a,b) into a merged image, the merged image including the image content of the areas of interest with a minimum fidelity. Monitoring arrangement (1) after claim 1 , characterized in that the processing unit (6) is designed to determine an image data stream (9) based on the image data (4), the image data stream (9) comprising the image data (4) in a quality-reduced, compressed and/or decoded manner, the Provision unit (15) is designed to provide the image data stream (9) to the central module (3), the combining unit (12) being designed to merge the image data stream (9) with the image content of the areas of interest to form the merged image. Monitoring arrangement (1) after claim 1 or 2 , characterized in that the client module (2) and/or the central module (3) has an evaluation unit, the evaluation unit being designed to determine additional information for the application-specific image data (8a, b), the additional information being a quality evaluation and/or relevance evaluation form and/or describe the image content of the areas of interest contained in the application-specific image data (8a,b), the combination unit (12) being designed to determine the merged image based on the additional information. Monitoring arrangement (1) after claim 3 , characterized in that the combination unit (12) is designed, in the case of redundancy and/or overlapping of areas of interest and/or image content of areas of interest of different application-specific image data (8a,b), the areas of interest and/or image content with a better quality rating and/or better relevance rating to be used to determine the fusion image. Monitoring arrangement (1) after claim 3 or 4 , characterized in that the quality assessment forms a numerical assessment, forms a peak signal-to-noise ratio and/or forms an equivalent bit/pixel ratio. Monitoring arrangement (1) according to one of the preceding claims, characterized in that a user can select an evaluation application (8a,b) as the preferred evaluation application (8a,b), the combination unit (12) being designed to combine image content and/or areas of interest to provide the application-specific image data (8a,b) of the preferred evaluation application (8a,b) in the merged image. Monitoring arrangement (1) according to one of the preceding claims, characterized in that the evaluation applications (8a, b) are each designed to determine a relevance map when executed and/or used as areas of interest, the relevance map assigning a plurality of pixels to image areas of the image data (4) forms. Monitoring arrangement (1) according to one of the preceding claims, characterized in that the application-specific image data (8a,b) include a reference to already transmitted and/or existing application-specific image data (8a,b). Monitoring arrangement (1) according to one of the preceding claims, characterized in that the evaluation applications (8a,b) are designed to apply uniform screening in image areas for detecting the areas of interest and/or areas of interest are subject to quantization of the uniform screening. Monitoring arrangement (1) according to one of the preceding claims, characterized in that the provision unit (15) is designed to monitor data traffic between the client module (2) and central module (3) and to determine a data traffic measure, the provision processing unit (15) is designed to manage the provision, transmission and/or retention of the application-specific image data (8a, b) based on the data traffic measure. Monitoring arrangement (1) according to one of the preceding claims, characterized by a feedback unit (16), the feedback unit (16) being designed based on the application-specific image data (8a, b) the processing unit (6) for generating the application-specific image data (8a, b) readjust, control and/or adjust. Method for monitoring a surveillance area (1), image data (4) being and/or being provided to a client module (2), a plurality of evaluation applications (8a,b) being executed and/or used on the client module (2), wherein the evaluation applications (8a,b) each detect areas of interest in the image data (4), application-specific image data (8a,b) being generated based on the interest data and the image data (4), the application-specific image data (8a,b) containing the image content of the Include areas of interest in a minimum fidelity, the application-specific image data (8a,b) being provided to the central module (3), a fusion image being determined on the central module (3) based on the application-specific image data (8a,b), the fusion image containing the image content which includes areas of interest in the high fidelity. Computer program, wherein the computer program is designed and/or set up according to the method during its execution and/or application claim 12 perform and/or apply. Machine-readable storage medium, with the computer program on the storage medium Claim 13 is saved.

Images