WO2022022809A1

WO2022022809A1 - Masking device

Info

Publication number: WO2022022809A1
Application number: PCT/EP2020/071259
Authority: WO
Original assignee: Robert Bosch Gmbh
Priority date: 2020-07-28
Filing date: 2020-07-28
Publication date: 2022-02-03

Abstract

Masking device (15) for a surveillance system (16), wherein the surveillance system (16) comprises at least one surveillance camera (9), wherein the surveillance camera (9) is arranged and/or configured to monitor a monitoring area (1) and to provide surveillance images (3), with a selection module (18) and a masking module (19), wherein the selection module (18) is configured for (10) selecting a masking area (8, 8a, b), wherein the masking area (8, 8a, b) is a subsection of the monitoring area (1), wherein the masking module (19) is configured to process the surveillance images (3) to edited images (10), wherein areas of the surveillance images (3) showing at least a part of the monitoring area (1) which is part of the masking area (8, 8a, b) are masked, wherein the (15) masking area (8, 8a, b) is defined by a location, a form and/or a size in three dimensions.

Description

Title

Masking device

Description

State-of-the-art

The invention is related to a masking device for a surveillance system wherein the surveillance system comprises at least one surveillance camera. The masking device is configured for editing surveillance images to edited images by masking.

Surveillance systems comprising cameras are widely used, for example in shops, airports, railway stations and also in public places like parks and streets. It is a serious concern of private people, companies and also of governments to protect the privacy of each person. For example, when a shop is monitored by a surveillance system people outside the shop, for example on the street, should not be shown in the monitoring images when the camera inside the room catches also pictures through the window. Therefore masking of public areas is needed.

For example the document DE 10 2008007 199 Al, which seems to be the closest state-of-the-art, discloses a module for masking selected objects in monitoring images. The monitoring area is captured in the monitoring images by a camera, wherein a user can select objects in those pictures, wherein the masking module is configured to mask those selected objects.

Disclosure of the invention

The invention discloses a masking device for a surveillance system with the features of claim 1. Furthermore, the invention concerns a surveillance system, a method, a computer program and a data carrier. Preferred and/or advantageous embodiments are shown in the description, the figures and the subclaims.

The invention concerns a masking device. The masking device is for a surveillance system and/or part of the surveillance system. The surveillance system comprises at least one surveillance camera, especially more than 10 surveillance cameras. The surveillance camera is for example a stereo camera. The surveillance camera can be a fixed camera with a fixed view or perspective or a movable camera with a movable view, for example pan- tilt-zoom camera. The surveillance camera is arranged and/or is configured to monitor a monitoring area. For example the surveillance camera takes images, also called pictures, of the monitoring area, whereby the images can be two-dimensional images or stereo images. The surveillance camera is configured to provide surveillance images, wherein the surveillance images showing especially the monitoring area as captured by the camera. The surveillance images are in particular unmasked. The surveillance camera may provide the surveillance images as an image stream, for example a video. Especially, when the surveillance system comprises more than one surveillance camera, surveillance images of this cameras are provided to the masking device. The masking device is for example coupled with the surveillance camera and/or the surveillance cameras, whereby the surveillance images are provided to the masking device.

The masking device comprises a selection module and a masking module. For example the selection module and the masking module are forming together a processing module. The selection module and/or the masking module may be a software module or a hardware model. In a preferred embodiment the selection module and/or the masking module comprises a neural network and/or adapted for machine learning. Preferably, the selection module and/or the masking module comprises a display for showing pictures. In particular, the selection module and/or the masking module comprise a touch screen. The selection module is preferably comprising or configured as a human-machine-interface, especially as a graphic user interface.

The selection module is configured for selecting at least one masking area. The selection of the masking area is maybe done by a person, especially called user. The masking area is for example a two-dimensional area, e.g. a rectangle or circle. Furthermore, the masking area is maybe a volume for example a cone. The selecting of the masking area may be done graphically by the user using the selection module and/or maybe done alpha-numerically. The masking area selected by the user is for example an area where objects, especially people, should not be shown in captured images. For example, the masking area is the area behind a shop window outside on the street. The masking area is a subsection of the monitoring area. Especially, the masking area is a subsection of the monitoring area which is shown in the surveillance images if there are no people in the scene.

The masking module is configured to process the surveillance images to edited images. For example the masking device is set between the surveillance camera and a data output for showing the surveillance on a display, whereby the masking device holds back the surveillance images and only allows to output edited images. The edited images are for example the surveillance images with masked parts. For example the masking module is configured to mask all parts of the surveillance image which are showing and/or belonging to the masking area. For example, all parts outside of the window, especially on the streets, are masked. For example, the masking module is checking if shown parts of the monitoring area in the surveillance images have an intersection with the masking area, whereby those areas with an intersection are masked.

The masking area is defined and/or selected by a location, form and/or a size in three dimensions. For example, the masking area comprises the information about its location, form and/or size in three dimensions. In particular, a coordinate system defined by the camera, for example based on its viewpoint, or a coordinate system based on the monitoring area is used to define and/or select the masking areas location, form and/or size. Especially, it is assumed that the masking area has a depth information and/or an information, e.g. the distance to the camera.

The invention is based on the idea that using a masking area defined in three dimensions can improve the information density which can be taken from a surveillance system, since less areas are unnecessarily and/or false masked. For example, current masking systems allow a user to draw masking areas onto the image or sensor plane, for example a rectangular shape. By drawing it on the image or sensor plane those masks yield in the 3-D monitoring area to a cone emerging from the camera viewpoint extending into infinity. This results in the problem that objects and/or people which are inside of the cone, especially between the plane and that viewpoint, are masked since they seem to be in the masking area. By using this invention the masking area is described in three- dimensions and/or yields in masking only objects and/or areas which are really inside the masking area and not only before it.

Preferably, the selection module is configured to allow a user to select the selected area in the surveillance image and/or based on the surveillance image. Especially, the selection of an area as selected area may be based on the edited images. For example, the selection module is configured as human machine interface, especially as a graphical user interface, whereby the user can select an area or volume as selected area based on the monitoring area in the images. For example, the selection module is configured to specify, define and/or compute the masking area based on the selected area and the depth information and/or distance information. For example, the selection module is configured to assign a depth information and/or distance information to the selected area. For example, the user can use the selection module to select a rectangular, polygonal or curved area as selected area, for example by drawing the area in a surveillance image.

The selection module may be configured for selecting a point or an area of the surveillance images as selection base. The selection module is for example configured to determine a selected area based on the selection base, for example using an image analysis and/or image or object classification. For example the user has selected a pixel or a group of pixels which belong to the same object, wherein the selection module is configured to set as selected area the whole object. The selection module may be configured to specify, define or determine the masking area based on the selected area, especially the selected base, and a depth information and/or a distance information. For example, the user may select a pixel or a group of pixel and set, for example type in or choose, the depth information and/or distance information, wherein the selection module is configured to define the selected area based on this input parameters.

Preferably, the selection module is configured to determine the selected area based on an object classification, for example the user is choosing a part of an object, for example part of a street, wherein the selection module is configured to select the whole object, for example whole street as selected area using and/or processing an object classification, for example object classification based on the surveillance image. Preferably, the object classification is adapted as a machine learning algorithm and/or using a neural network.

Preferably, the selection module is configured to allow a user to enter and/or select the depth information and/or distance information. The user may type in the distance and depth information. Alternatively, the selection module is configured to allow the user to select and/or enter the depth and/or distance information from a table, sliding bar or graphically. For example, the user knows that the window, which should be set as selected area and/or as masking area, is 3 m away from the camera so that the user can enter and/or select this number as distance and/or depth information.

In a preferred embodiment of the invention the selection module is configured to determine the depth information and/or the distance information based on a stereographic analysis of the image, especially the surveillance image. For example, the user has selected an area and/or a selection based object, wherein the selection module is configured to check how far away the object and/or part of the monitoring area belonging to this selection is by using a stereographic analysis. For example, the surveillance camera is a stereoscopic adapted camera and provides a pair of pictures usable for stereographic analysis. This embodiment is based on the idea to provide a very easy handling masking device, wherein the user only has to choose an area without knowing how far this object is away from the camera, wherein the selection module is configured to determine the distance or depth information by itself.

Optionally, the masking module is configured to apply an image analysis to the surveillance images. The image analysis is configured to obtain image portions. The image portions are for example parts of the image that can concern to the same object, have the same colour or design. The masking module is configured in this embodiment to mask the image portions that are a part of the masking area and/or have an intersection with the masking area.

Especially, the masking module is configured to obtain an image portion depth to each image portion. Especially the masking module is configured to mask the image portions based on the image portion depth, for example by checking if the image depth is behind and/or a bigger than the depth of the masking area.

For example the image analysis is configured as object detection and/or object classification, for example the image analysis is configured to obtain image portions which belonged to a same object for example to obtain people in the surveillance image. Preferably an image portion contains all pixels and/or image parts that belong to this and/or one object. The masking module is for example configured to obtain for the image portion which belongs to the and/or one object the distance, for example average distance, of the object from the camera and use and/or set this distance as image portion depth. This embodiment is based on the idea that using the surveillance image to classify and/or detect objects and determine the distance and/or depth of this object, whereby by comparing the distance of this object with the distance and/or the location of the masking area objects that are nearer to the camera than the masking area are still shown as object and are not masked.

The masking module can be configured to obtain the image portion depth based on a stereoscopic image analysis, for example when the surveillance camera is configured as a stereoscopic camera. Alternatively and/or additionally the masking module is configured to obtain the image portion depth based on an object tracing and/or tracking for example an object which is known to be inside a store and cannot be outside in the street may be traced and/or tracked, whereby by knowing that the object or person had not used the door this object and/or person has still to be before the masking area. Furthermore, the masking module may be adapted and/or configured to use a physical sensors to obtain the image portion depth, for example a radar sensor or an infrared sensor. The physical sensor is preferably part of the surveillance system, for example part of the surveillance camera.

A further object of the invention is a surveillance system. Particularly, the surveillance system comprises at least one surveillance camera to monitor the monitoring area and to provide surveillance images. Furthermore, the surveillance system comprises the masking device as previous described. Especially, the surveillance camera is connected with the masking device to provide the surveillance images to the masking device. Furthermore, the invention concerns a method for masking surveillance images. The method contains selecting of a masking area, wherein the masking area is a subsection of the monitoring area and/or a subsection of the surveillance images. Particularly, the method is executed and/or used by the masking device. According to the method, parts of surveillance images are masked, wherein parts of the surveillance images which are part and/or have an intersection with the masking area are masked. The masked images are called edited images, where by the edited images are especially provided to a monitoring center.

Another object of the invention is computer program, wherein the computer program is adapted to run the method, especially when the computer program is run on a computer and/or a masking device.

Further advantages and/or preferred embodiments are shown in the figures and its description. Thereby,

Figure la and lb shows a surveillance scene and masks;

Figure 2 scene and mask in top view from Figures 1;

Figure 3 masking device.

Figure la shows a monitoring area 1 which is monitored via a surveillance camera of the surveillance system. The monitoring area 1 is part of the shop, whereby sales objects 2 are presented for a selling. The surveillance camera 9 is taking surveillance images 3 of the monitoring area 1. The shop comprises a window 4 and a door 5, whereby the window 4 and the door 5 are made of glass and hence public areas 6 outside the shop are also captured by the surveillance images 6. Since a general interest is the protection of privacy and the right of a person on its own picture, a device for masking the public areas 6, and hence the window 4 and the door 5, is needed.

Figure lb shows an edited image 10 based on the surveillance image 3 of figure la, whereby the public areas 6 are masked by a mask 7a and 7b. The masks 7a and 7b are covering the masking areas 8a, 8b. The masking areas 8a, 8b are defined in three dimensions which means that the masking areas 8a, b are defined by an extension in the image plane and a distance to the surveillance camera 9. Therefore, parts in the public areas 6 outside the window and outside the door are covered by the mask 7a, b, whereby parts inside the shop are still shown in the edited image 10.

Figure 2 shows a top view of the monitoring area 1 of figure la, b. For clarity reasons the selling objects 2 are not shown in the figure 2. The masking areas 7a, b comprises distance information, also called depth information, whereby the distance information describes the distance of the masking area 7a, b from the camera 9. The distance information is for example defined by a coordinate system 11 of the real world.

The camera 9 is configured and arranged to capture this part of the store which is inside the detection zone 12 of the camera 9. Without defining the masking areas 8a, b in three dimensions, for example like in the state-of-the-art defining chest and the image plane, parts 13a, b of the monitoring area 1 between the masking areas 8a, b and the camera 9 would also be covered by the mask 7a, b. For example a person 14 in the area 13b would not be shown in the edited images 10 and covered by the mask 7b. By defining the masking areas 8a, b in three dimensions and applying a depth information and/or distance information to it, objects between the masking areas 8a, b, like the person 14, would not be covered by a mask since the distance information and/or depth information of the object or person 14, differs from the depth information and/or distance information of the masking area 8a, b.

Figure 3 shows an example of masking device 15. The masking device 15 is placed between the surveillance system 16, comprising several surveillance cameras 9, and a monitoring centre 17 for displaying and showing the edited images 10. The surveillance system 16, especially the surveillance cameras 9, are providing the surveillance images 3 to the masking device 15. The masking device 15 comprises selection module 18 which is configured to select one or more masking areas 8. For example the selection module comprises a graphical user interface, whereby the user can select parts of the monitoring area 1 as masking areas 8. The selection module 18 is connected with the masking module 19 for providing the masking areas 8 to the masking module 19.

The masking module 19 is configured to cover those parts of the surveillance image 3 which show these parts of the monitoring area 1 that are part of the masking area 8. The masking module 19 provides the masked images as edited images 10 to the monitoring 17. Therefore surveillance images 3 which contain private areas or public areas, like streets, are not provided to the monitoring centre 17, such that privacy interests and the right on the own picture are ensured.

Claims

1. Masking device (15) for a surveillance system (16), wherein the surveillance system (16) comprises at least one surveillance camera (9), wherein the surveillance camera (9) is arranged and/or configured to monitor a monitoring area (1) and to provide surveillance images (3), with a selection module (18) and a masking module (19), wherein the selection module (18) is configured for selecting a masking area (8, 8a, b), wherein the masking area (8, 8a, b) is a subsection of the monitoring area (1), wherein the masking module (19) is configured to process the surveillance images (3) to edited images (10), wherein areas of the surveillance images (3) showing at least a part of the monitoring area (1) which is part of the masking area (8, 8a, b) are masked, wherein the masking area (8, 8a, b) is defined by a location, a form and/or a size in three dimensions.

2. Masking device (15) according to claim 1, wherein the selection module (18) is configured to allow a user to select in the surveillance pictures (3) an area as selected area, wherein the selection module (18) is configured to specify the masking area (8, 8a, b) based on the selected area and a depth and/or distance information.

3. Masking device (15) according to claim 1 or 2, wherein the selection module (18) is configured to allow a user to select a point or area of the monitoring images (3) as selection base, wherein the selection module (18) is configured to determine a selected area based on the selection base and an image analysis, wherein the selection module (18) is configured to specify the masking area (8, 8a, b) based on the selected area and a depth and/or distance information.

4. Masking device (15) according to claim 3, wherein the selection module (18) is configured to determine the selected area based on an object classification.

5. Masking device (15) according to one of the claims 2 to 4, wherein the selection module (18) is configured for entering and/or selecting the depth and/or distance information by the user.

6. Masking device (15) according to one of the claims 2 to 5, wherein the selection module (18) is configured to determine the depth and/or distance information based on a stereographic analysis of the surveillance pictures (3).

7. Masking device (15) according to one of the previous claims, wherein the masking module (19) is configured to apply an image analysis to the surveillance images (3) to obtain image portions, wherein the masking module (19) is configured to mask image portions which are part of the masking area (8, 8a, b).

8. Masking device (15) according to claim 7, wherein the masking module (19) is configured to obtain an image portion depth for each image portion and mask based on the image portion depth image portions which are behind the masking area (8, 8a, b).

9. Masking device (15) according to claim 8, wherein the masking module (19) is configured to obtain the image portion depth based on a stereoscopic image analysis, an object tracing and/or a physical sensor of the surveillance system.

10. Surveillance system with at least one surveillance camera and a masking device according to one of the previous claims.

11. Method for masking surveillance images (3), wherein a masking area (8, 8a, b) is selected, wherein the masking area (8, 8a, b) is defined by a location, a form and/or a size in three dimension, where in the surveillance images are showing a monitoring area, wherein the surveillance images (3) are processed to edited images (10) by masking areas of the surveillance images (3) showing at least a part of the monitoring area (8, 8a, b).

12. Computer program, wherein the computer program is configured to execute the method according to claim 11.

13. Data carrier, wherein the computer program according to claim 12 is stored on the data carrier.