DE102018204704A1 - System for monitoring a surveillance area - Google Patents

Abstract

A system and a method for monitoring a monitoring area include a plurality of imaging sensors for detecting a movement behavior of at least one movable object in the surveillance area, an evaluation device for calculating a trajectory of the at least one movable object as a function of the detected movement behavior and an interface for making available the calculated Trajectory for controlling an operation in the surveillance area, wherein at least one imaging sensor of the plurality of imaging sensors is arranged on a mobile sensor carrier.

Description

Technical part

The invention relates to a system for monitoring a monitoring area and an associated method. In particular, the invention relates to a system and a method in which a plurality of imaging sensors are used to detect a movement behavior of at least one movable object in the surveillance area.

State of the art

Systems with imaging sensors for detecting objects in a surveillance area are known, and with such systems, locations of objects can be determined.

In the US 2015/0012396 A1 For example, a stationary imaging system is described for monitoring inventory. The imaging system determines whether a warehouse worker has taken a stored object from a monitored storage location or has placed an object in a storage location. An operationally relevant inventory turnover can thus be determined and evaluated temporarily.

However, operations are not only impacted or reconciled with operational locations of objects, but are also dynamically affected by object movement within an area relevant to operations.

Summary of the invention

The invention therefore provides solutions for efficiently detecting object movements in a region to be monitored in order to dynamically optimize operations.

Such a solution for a system for monitoring a monitoring area has a plurality of imaging sensors for detecting a movement behavior of at least one movable object in the surveillance area, an evaluation device for calculating a trajectory of the at least one movable object as a function of the detected movement behavior and an interface for making available calculated trajectory for controlling an operating sequence in the surveillance area, wherein at least one imaging sensor for detecting the movement behavior of the at least one movable object is arranged on a mobile sensor carrier.

The system may be configured to combine the calculated object trajectory with static and georeferenced map data to provide a dynamic map or dynamic geographic information system for controlling an operation. For this purpose, the calculated object trajectory can be transformed into a coordinate system of the map or the geoinformation system.

A monitoring area can be a factory, a production facility or a warehouse, which can each be found in particular in a hall.

An object located in the surveillance area and to be monitored may be an object, a material or a person, wherein the movement behavior of the object may be a movement of the object, a material movement or a person movement or a corresponding flow or current. The movement behavior may include object positions, an object speed, an object acceleration, and / or a residence time of the object at an object position.

An imaging sensor may be any sensor that is configured to acquire two-dimensional or three-dimensional image information or image data, wherein the acquired image information may be pixel- or voxel-based information that may have coordinates on the one hand and color-coded information on the other. The imaging sensor can also be a photogrammetric sensor, in particular a camera, an optical sensor and / or a scanning sensor. In addition to one or more mobile imaging sensors, in particular at least one stationary imaging sensor can be provided.

The mobile sensor carrier may be a vehicle, a facility, a robot, or a person-carried carrier. The vehicle may be a person-operated or autonomous vehicle, for example an industrial truck. The system can be a stationary or mobile operating system, for example a crane.

The trajectory of the object can have location-related information which makes it possible to describe an object movement or a change in position of the object. Two positions or a vector can already describe the object movement. The trajectory can also have or be based on several object positions, wherein the individual positions can be connected in a linear or curved manner or determined by means of interpolation.

The interface can be provided for a data transfer of the calculated trajectory, wherein the location-related and / or geometric information of the trajectory can be taken into account when controlling an operating procedure. The operating process to be controlled can be, for example, material-based, logistical, production-dependent, building-technical or personnel processes for the operation of a production facility, a factory or a warehouse.

The trajectory provided at the interface has the effect that when controlling an operation, not only static object information but also dynamic object information can be accessed to take account of recent, current, or future object movement, and thus optimize operation. The dynamic object information may include temporary and kinematic information. Thus, the movement behavior of monitored objects in an existing material flow system or logistics system, in a production control, in a building control or in a personnel management system can be taken into account.

A core idea of the invention can be seen in that movements of objects with a multiplicity of imaging sensors, that is to say with at least two such sensors, which are sometimes or predominantly arranged on mobile sensor carriers, are detected in the near real time with high monitoring coverage to realize. The provision of a mobile imaging sensor may be advantageous for substantially gap-free area monitoring because the detection range of such a sensor may be more variable and larger than a stationary imaging sensor, and thus the required number of imaging sensors may be reduced.

The invention is also based on the concept that the movement behavior of dynamic objects can be continuously monitored or tracked by the use of mobile image acquisition systems, without the dynamic objects having their own sensors or object components, which actively communicate with a sensor system, being equipped for this purpose.

In one embodiment, the plurality of imaging sensors comprise a plurality of cameras for continuously acquiring image information, wherein at least one camera of the plurality of cameras is disposed on a mobile camera carrier. A camera may be a digital camera for capturing static or moving images, which may be a video camera. Continuously acquired image information may be captured at specific intervals. The object to be monitored can be imaged in the image information of the individual images, it being possible to detect and calculate an object movement from a comparison of a plurality of individual images by means of a difference image. For calculating relative or absolute object coordinates or object motions, it may be necessary to determine a position, orientation and calibration of the camera. Continuously determining a camera position and a camera orientation may be required when the camera is on the mobile camera carrier. Furthermore, it may be necessary to transform an object movement in image coordinates into an absolute object movement in a superordinate coordinate system.

Recognition of the object to be monitored in the acquired image information can take place with known automatic methods of image analysis, wherein matching with known templates can be carried out in order to monitor the movement behavior of predefined objects. An automatically recognized object can thus be assigned to a predefined or known object.

In a further embodiment, the mobile sensor carrier is designed to carry the at least one imaging sensor and the movable object. The mobile sensor carrier may also be a mobile slide. If the moveable object is located on the mobile sensor carrier, wherein the moveable object can be carried and moved by the mobile sensor carrier, the imaging sensor can continuously acquire image information of the object environment. The object itself can not be displayed in the image information. From several individual images of the environment and from a comparison of the individual images, the movement of the mobile sensor carrier can be calculated, from which also the trajectory of the object moved by the sensor carrier can be calculated. The trajectory of the object can thus be determined indirectly from image information of its surroundings, which are acquired by different object positions.

In a further embodiment, the mobile sensor carrier is an autonomous vehicle. On the autonomous vehicle, an environment detection system may be arranged, which may have the at least one mobile imaging sensor for detecting the movement behavior of the object. The autonomous vehicle may be a surveillance vehicle intended to monitor the surveillance area. Furthermore, the autonomous vehicle may be a vehicle used to transport objects in the surveillance area is provided. Alternatively or additionally, the vehicle can also carry out other work, in particular work, which are related to the operating procedure to be controlled in the monitoring area. Based on the calculated trajectory of objects in the vicinity of the vehicle can also be intervened in the control of the vehicle and, for example, to stop or modify a vehicle movement in order to avoid a collision with the monitored object.

In a further embodiment, the mobile sensor carrier is carried by a person. The person may carry a portable camera on a garment, work equipment or helmet. The camera can be a body cam, which can be arranged on the body or on a worn work equipment. Furthermore, an object to be monitored can also be moved by a person in the surveillance area.

In a further embodiment, the evaluation device is configured to transform the calculated trajectory into a dynamic map, and an information system based on the dynamic map has the interface that is set up to make available the transformed trajectory for controlling an operating procedure in the surveillance area. The dynamic map may be based on a static map that can be overlaid with dynamic map information. Such a dynamic map may include and display information about the movement behavior of the objects to be monitored. The monitored objects can thus be displayed as dynamic objects in the map. With the trajectory transformed into the map, past, current and future position information of the objects can be visualized and analyzed.

Images captured by the imaging sensors can also be used to generate static map data of the dynamic map. For this purpose, several captured individual images can be merged, that is, combined with each other. Combining multiple frames can be done by image stitching. The result may be a photorealistic representation or map of the surveillance area in which moving objects can be dynamically displayed.

A dynamic map based information system may be a standalone or dynamic geographical information system (GIS) associated with a system for controlling an operation, which may also maintain dynamic map data about object motion based on geo-referenced static map data. This may be, for example, a factory information system, which can provide movement data of material to be processed and workers independently or other systems for factory control, for example a shop floor management system.

The calculated trajectory can, for example, also be integrated directly into the control of an automated guided vehicle system (AGV). This has the advantageous effect that the monitoring of object movements and their trajectory in the operation of driverless transport systems can avoid collisions of the transport systems with the monitored objects and can enable route re-planning.

In a further exemplary embodiment, at least one imaging sensor of the plurality of imaging sensors is located in a partial area of the monitoring area, which represents the potential location of the object as a function of a modeled movement behavior of the object. For the presence of the sensor in the subarea, the mobile sensor carrier, on which the sensor can be arranged, can be moved into the subarea for this purpose. The future probable movement behavior of the object can be predicted by extrapolation of an already calculated trajectory or by probability calculation. For this purpose, the movement behavior of objects can also be learned and imaged by means of physical or static models or by means of a neural network.

In a further embodiment, a data source for metadata of the movable object is provided, wherein the metadata of the trajectory of the object are assignable. The metadata may include additional object data or data about the trajectory of the object.

In a further embodiment, the plurality of imaging sensors are in the vicinity of the movable object. The imaging sensors may be all or part of the surveillance area. In other words, no imaging sensor or an element actively communicating with a sensor can be located on the object to be monitored itself. The object can thus behave passively when detecting its movement behavior. It does not actively participate by directly providing information on the detection of its movement behavior.

In a further embodiment, the mobile sensor carrier has additional non-imaging sensors for detecting a movement behavior of the movable object. Is the object to be monitored on the sensor carrier, can a trajectory of the object are optionally determined for an image-based evaluation by sensors on the mobile sensor carrier, which detect a position, an orientation, a distance, a speed or an acceleration of the mobile sensor carrier. The trajectory of the mobile sensor carrier can thus also be used to deduce the trajectory of the object to be monitored if the object is moved by the mobile sensor carrier.

Alternatively or additionally, for determining the position of the mobile sensor carrier, several reference points can also be accessed, to which the distance from the sensor carrier can be determined by measurement, in order to determine the position, for example, by means of a spatial rearward section.

Another solution is a method for monitoring a surveillance area, which comprises detecting a movement behavior of at least one movable object in the surveillance area with a multiplicity of imaging sensors, calculating a trajectory of the at least one movable object depending on the detected motion behavior and making the calculated trajectory available for controlling an operating sequence in the monitoring area, wherein the detection of the movement behavior of the at least one movable object is carried out by at least one imaging sensor which is arranged on a mobile sensor carrier.

The invention and its embodiments are further explained in the attached schematic figures.

list of figures

• 1 shows an embodiment of the system for monitoring a surveillance area.
• 2 FIG. 13 is a diagram for explaining detection of a movement behavior of an object with an surroundings detection system of a vehicle when monitoring the surveillance area. FIG.
• 3 is another illustration for explaining a detection of a movement behavior of an object with an environment detection system of a vehicle when monitoring the monitoring area.
• 4 FIG. 10 is a diagram for explaining detection of a movement behavior of a vehicle moving object with an surroundings detection system of the vehicle in monitoring the surveillance area. FIG.
• 5 FIG. 8 is another diagram for explaining detection of a movement behavior of a vehicle moving object with an surroundings detection system of the vehicle in monitoring the surveillance area. FIG.
• 6 is a representation for explaining a continuous Umweisfassung along a vehicle trajectory.
• 7 FIG. 10 is a flowchart of method steps of one embodiment of a method for monitoring a monitored area. FIG.

Detailed description of embodiments

In 1 is a surveillance system 200 to monitor an object 4 in a surveillance area 6 shown. System components for object monitoring are a variety of themselves in the surveillance area 6 located cameras 10 . 20 , an autonomous vehicle 22 as a mobile camera carrier for a mobile camera 20 , an evaluation device 40 for calculating an object trajectory from the cameras 10 . 20 captured image information and interfaces 210 to provide the calculated object trajectory for other systems 212 . 214 . 216 . 218 ,

In the surveillance area 6 are next to the on the autonomous vehicle 22 arranged mobile camera 20 more mobile cameras 20 on a person 24 and a mobile camera 20 on a hall crane as a mobile plant 28 intended. The mobile cameras 20 on the person 24 are provided as a body cam on the body and as a helmet cam on the helmet. In addition to the mobile cameras 20 is also a stationary camera 10 in the surveillance area 6 firmly arranged. The cameras 10 . 20 capture image information 8th in their respective detection area, which by the fixed angle of the stationary camera 10 and the variable viewing angles of the mobile cameras 20 is fixed.

The cameras 10 . 20 are via a stationary transmitting and receiving unit 34 with the interfaces 210 wirelessly connected via WLAN or wired via LAN and communicate with them accordingly (connection partially not shown). The mobile camera 20 on the autonomous vehicle 22 is with a mobile sending and receiving unit 32 on the autonomous vehicle 22 connected to the stationary transmitting and receiving unit 34 wirelessly connected. From the cameras 10 . 20 captured image information 8th is via the transmitting and receiving units 32 . 34 so to the evaluation unit 40 transfer.

As monitored objects 4 are in 1 on the autonomous vehicle 22 transported objects 4 shown, whose movement behavior is at least from that of the on the autonomous vehicle 22 arranged camera 20 captured image information 8th lets calculate in which a trajectory 5 of the autonomous vehicle 22 by means of the captured image information 8th and / or by means of an environment detection system 30 which includes at least one further environment detection sensor 36 has calculated. The at least one further environment detection sensor 36 may be exemplified as a radar sensor, lidar sensor, infrared sensor or ultrasonic sensor. The calculation of the trajectory 5 done with the evaluation 40 ,

The calculated trajectory 5 is over the interfaces 210 the systems 212 . 214 . 216 . 218 provided to which system users 202 access information about the movement behavior of the objects 4 to preserve or to the movement behavior of the objects 4 continue to evaluate. As other systems are a logistics system 212 , a production control system 214 , a control system for driverless transport systems 216 and / or a facility management system 218 intended.

The other systems bind the from the evaluation unit 40 calculated trajectory 5 the objects 4 and the autonomous vehicle 22 in operations that you control. Controllable operations of the logistics system 212 are managing logistics orders, controlling the autonomous vehicle 22 , a status capture of the objects 4 along a logistics chain and / or inbound and outbound coverage of objects 4 or of persons 24 , Controllable operations of the production control system 214 are a control of machine tools or other equipment and / or performing a quality inspection for objects to be processed 4 , Controllable operations of the control system for automated guided vehicles 216 are navigating autonomous vehicles 22 as well as a loading and unloading of the autonomous vehicles 22 with objects 4 , whereby also a Kollisionsvermeiden with persons 24 is controllable. Controllable operations of the facility management system 218 are a control of ventilation, air conditioning, lighting, access control and time detection depending on the monitored movement behavior of objects 4 in the surveillance area 6 ,

In the 2 to 5 is a dynamic map 100 a warehouse as a surveillance area 6 shown. The dynamic map has static mapping 110 of the camp. In the surveillance area 6 there is an object 2 for example, a worker or a pallet, and a storage object 4 their movement behavior from the monitoring system 200 is monitored.

As in 2 is shown, image information 8th in the environment of the object 2 detected, wherein in the captured image information 8th the object 2 is shown. The captured image information 8th becomes exemplary with the mobile camera 20 on the autonomous vehicle 22 detected. As in 3 is shown, further image information 8th in the environment of the object 2 captured, with the object 2 again in the captured image information 8th is shown. A trajectory 5 as movement behavior of the object 2 gets out of the picture information 8th calculated and as dynamic map information 120 in the dynamic map 100 shown.

Is the object lying 2 no longer in the detection range of the mobile camera for a predetermined period of time 20 of the autonomous vehicle 22 , then the last detected position of the object 2 , as in 4 represented as static map information 122 shown. According to 4 became the storage object 4 already taken from its storage location and on the autonomous vehicle 22 stored. Then picture information 8th in the environment of the storage object 4 detected, wherein in the captured image information 8th not the storage object 4 himself, but his environment. The captured image information 8th is with the mobile camera 20 on the autonomous vehicle 22 captured after the warehouse object 4 on the autonomous vehicle 22 is located and moved by this. Moves the autonomous vehicle 22 but still on the storage object 4 to, then becomes the storage object 4 from the mobile camera 20 of the autonomous vehicle 22 detected. About the captured image information 8th will also be on stock object 4 attached identification of the storage object 4 , such as a label, a number or a bar code, via which further object data or object information of the storage object 4 become available. This the warehouse object 4 characterizing object data or object information can be sent to the evaluation unit 40 transmitted and with existing object data or object information in the systems 212 . 214 . 216 . 218 be checked for consistency by comparison. If there is a discrepancy between the captured and existing object data or object information, those in the systems 212 . 214 . 216 . 218 existing object data via the interfaces 210 be updated or supplemented accordingly.

In 5 will be more picture information 8th the environment of the storage object 4 captured, with the object 4 again not in the captured image information 8th is shown. A trajectory 5 as movement behavior of the bearing object 4 gets out of the picture information 8th of the environment along that of the autonomous vehicle 22 crazy trajectory 5 calculated and as dynamic map information 120 in the dynamic map 100 shown.

One of the systems 212 . 214 . 216 . 218 so can about the current positions and trajectories 5 the objects 2 . 4 being informed, for example, an arrival time of one of the objects 2 . 4 or a location of the objects 2 . 4 is used to control an operation.

In 6 is a continuous environment detection along a trajectory 5 an autonomous vehicle 22 shown. With an environment detection system 30 which has several cameras 20 has surrounding images 60 detected by means of features recognized in the environment images 61 to an overall picture 62 be merged. The overall picture 62 can be called static mapping 110 and / or for calculating dynamic map information 120 in the dynamic map 100 be used. From the fusion of environmental images 60 is also the trajectory 5 of the autonomous vehicle 22 or objects 4 , which is based on the autonomous vehicle 22 are calculable. The trajectory 5 is optionally also from other sensors (not shown), for example an odometer or acceleration sensors, on the autonomous vehicle 22 calculable or the derivative of the trajectory 5 from the environment pictures 60 is sustainable with measurement data of such sensors, for example by means of SLAM method.

With the in 1 shown monitoring system 200 and in the 2 to 6 The automatic surrounding detection methods shown become the movement behavior of objects 2 . 4 calculated and other systems 212 . 214 . 216 . 218 for controlling operations in a surveillance area 6 provided.

In 7 are a flow chart for monitoring a monitored area 6 as process steps a first step S1 which detects a movement behavior of an object 2 . 4 has a second step S2 which calculates a trajectory 5 of the object 2 . 4 and a third step S3 which makes the trajectory available 5 for other systems 212 . 214 . 216 . 218 has shown.

LIST OF REFERENCE NUMBERS

2

object

4

warehouse

5

trajectory

6

monitoring area

8th

image information

10

stationary camera

20

mobile camera

22

autonomous vehicle

24

person

28

mobile plant

30

Environmental detection system

32

mobile transmitting and receiving unit

34

stationary transmitting and receiving unit

36

Environment detection sensor

40

evaluation

60

Umgebungsbild

61

feature

62

overall picture

100

dynamic map

110

static mapping

120

dynamic map information

122

static map information

200

monitoring system

202

system users

210

interface

212

logistics system

214

Production control system

216

Control system of driverless transport systems

218

Facilities Management System

S1

Capture movement behavior

S2

Calculate trajectory

S3

Exposing trajectory

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 2015/0012396 A1 [0003]

Claims (11)

A system for monitoring a surveillance area (6), comprising a plurality of imaging sensors for detecting a movement behavior of at least one movable object (2, 4) in the surveillance area (6), an evaluation device (40) for calculating a trajectory (5) of the at least one movable object (2, 4) as a function of the detected movement behavior and an interface (210) for making available the calculated trajectory (5) for controlling an operating sequence in the monitoring area (6), characterized in that at least one imaging sensor for detecting the movement behavior of the at least one movable object (2, 4) is arranged on a mobile sensor carrier. System after Claim 1 characterized in that the plurality of imaging sensors comprises a plurality of cameras (10, 20) for continuously acquiring image information and at least one camera (20) of the plurality of cameras (10, 20) is disposed on a mobile camera carrier. System after Claim 1 or 2 , characterized in that the mobile sensor carrier for carrying the at least one imaging sensor and the movable object (2, 4) is formed. System according to one of the preceding claims, characterized in that the mobile sensor carrier is an autonomous vehicle (22). System according to one of the preceding claims, characterized in that the mobile sensor carrier by a person (24) is carried. System according to one of the preceding claims, characterized in that the evaluation device is arranged to transform the calculated trajectory into a dynamic map (100) and an information system based on the dynamic map (100) has the interface (210) which makes available the transformed map Trajectory is set up for the control of an operation in the monitoring area. System according to one of the preceding claims, characterized in that there is at least one imaging sensor of the plurality of imaging sensors in a partial area of the monitoring area, which represents a potential location of the object (2, 4) depending on a modeled movement behavior of the object. System according to one of the preceding claims, characterized in that a data source is provided for metadata of the movable object, wherein the metadata of the trajectory (5) of the object (2, 4) are assignable. System according to one of the preceding claims, characterized in that the plurality of imaging sensors in the environment of the movable object (2, 4) are located. System according to one of the preceding claims, characterized in that the mobile sensor carrier additional non-imaging sensors for detecting a movement behavior of the movable object (2, 4). A method for monitoring a surveillance area, comprising the steps of detecting (S1) a movement behavior of at least one movable object (2, 4) in the surveillance area (6) with a plurality of imaging sensors, calculating (S2) a trajectory (5) of the at least one movable one Object as a function of the detected movement behavior and making available (S3) the calculated trajectory (5) for controlling an operation in the surveillance area, characterized in that detecting (S1) the movement behavior of the at least one movable object of at least one imaging sensor, which on a is arranged mobile sensor carrier is performed.

Images