DE102019127826B4 - Safe optoelectronic sensor and method for securing a surveillance area - Google Patents

Abstract

Safe optoelectronic sensor (10) for securing a monitoring area (12) with at least one machine (24), in particular a robot, the sensor (10) having an image sensor (14) with a large number of pixel elements for generating image data of objects (24, 26) in the monitoring area (12) and a control and evaluation unit (18) which is designed to use the image data to detect objects (24, 26) and to at least determine and evaluate their position in order to prevent danger to act on the machine (24), characterized in that the image sensor (14) is an event-based image sensor, which by means of two-channel or self-checking architecture and / or by plausibility checks of the image data by means of the control and evaluation unit (18) as a safe event-based image sensor (14 ) is designed, and that the control and evaluation unit is designed to include events detected by the pixel elements as moving objects to treat.

Description

The invention relates to a safe optoelectronic sensor, in particular a safe camera, and a method for protecting a surveillance area according to the preamble of claims 1 and 14, respectively.

The primary goal of safety technology is to protect people from sources of danger, such as those presented by machines in an industrial environment. The machine is monitored with the help of sensors, and if there is a situation in which a person threatens to get dangerously close to the machine, a suitable safety measure is taken. The current protection concept provides that protective fields are configured that are not allowed to be entered by the operating personnel while the machine is in operation. If the sensor detects an impermissible protective field intervention, for example an operator's leg, it triggers a safety-related stop of the machine.

Sensors used in safety technology must work particularly reliably and therefore meet high safety requirements, for example the EN13849 standard for machine safety and the IEC61496 or EN61496 device standard for electro-sensitive protective devices (ESPE). A number of measures must be taken to meet these safety standards, such as safe electronic evaluation through redundant, diverse electronics, function monitoring or, in particular, monitoring of the contamination of optical components.

With the increasing use of robots or so-called cobots (collaborative robots) in the industrial sector, safety aspects and human-robot collaboration (HRC) are becoming more and more important. Relevant standards in this context are, for example ISO 10218 for industrial robots or the ISO / TS 15066 for collaborative robots. By definition, HRC requires working very close to the robot. Protective fields and safety distances should therefore remain as small as possible, of course subject to the stipulation that safety remains guaranteed. The norms ISO13854 , ISO 13855 , ISO 13857 deal with the definition of safety distances. Since the threat to which people are exposed changes almost continuously, highly dynamic, situation-adapted security concepts are also required. The collaboration requires the safety function to be requested very frequently.

An evaluation of objects and machines in terms of speed and mutual distance is referred to as "Speed and Separation Monitoring" in the robot standards mentioned. Most conventional safe monitoring sensors, such as laser scanners or 3D cameras, do not support this. They continue to work with the usual protective fields, and they only provide a binary switch-off signal to indicate whether a protective field is violated or not. Security is guaranteed in this way, but not particularly flexible and often with unnecessary security surcharges and thus restrictions on availability, since the protective fields are configured for worst-case scenarios and not the actual current situation.

There are also some approaches in the prior art for monitoring the distances between the machine and an object that is possibly too close. For example, stereo cameras or time-of-flight cameras are known which generate and output corresponding distance information. Even with conventional image cameras that deliver two-dimensional images, a correspondingly complex image evaluation can be used to ensure a safe environment. Thermal cameras in particular can differentiate between living beings and objects.

A disadvantage of image cameras is that changes are recorded by comparing two or more images. This requires complex image evaluation and therefore high computing power. Another disadvantage is that the information is discrete due to the frame rate. What happens between the frames is not measured. A higher flow of information could be achieved with high-speed cameras, but that only works to a limited extent, and in addition to the high hardware costs there is the even more complex processing of the immense amounts of data. There are also algorithms that determine the optical flow, for example, and thus predict what will happen between the frames or in future frames. This in turn means a high computational effort and generates further errors, since the real measurement information remains the same.

Recently, a new type of camera technology has emerged, the so-called event-based or event-based camera. Based on the visual cortex, it is also referred to as a neuromorphological camera. So far, however, this has not been brought into connection with safety technology or even human-robot cooperation. A conventional camera exposes all of its pixels at a regular frame or image repetition rate and then reads them out simultaneously in order to obtain a number of stationary two-dimensional images per observed period corresponding to the image repetition rate.

In an event-based camera there is also a matrix of pixels, but neither a fixed frame rate nor a common one Reading out pixels. Instead, each pixel checks individually to see if it detects a change in intensity. Only in this case is image information output or read out, and only from this pixel. Each pixel is thus a kind of independent motion detector. A detected movement is reported individually as an event. As a result, the event-based camera reacts extremely quickly to the dynamics in the scene. The image data cannot be grasped so intuitively for the human eye because the static image components are missing.

An event-based camera is described, for example, in a whitepaper from the Prophesee company, which can be found on their website.

From the WO 2015/036592 A1 , WO 2017/174579 A1 and WO 2018/073379 A1 pixel circuits for an event-based camera are known.

The work of Gallego, Guillermo, Henri Rebecq, and Davide Scaramuzza, "A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation", IEEE Int. Conf. Comput. Vis. Pattern Recog. (CVPR), Vol. 1. 2018 presents methods of how movement, distances and optical flow can be determined from the data from an event-based camera.

The US 2013/0 335 595 A1 discloses a device and a method for event-based image processing, specifically for determining the optical flow, which, however, as already mentioned, is a very complex and expensive tool.

The DE 10 2013 106 514 B3 discloses securing an automated machine that is monitored by a camera system. When a foreign object is detected in a configured protection area, a safety-relevant function is triggered, unless it is determined by analyzing the camera images that the foreign object is a welding spark.

In the DE 10 2017 207 852 A1 describes a fire detection device that uses a two-dimensional dynamic vision sensor.

Against this background, the object of the invention is therefore to improve the safe optical monitoring of a machine.

This object is achieved by a safe optoelectronic sensor, in particular a safe camera, and a method for securing a surveillance area according to claims 1 and 14, respectively. The sensor is safe, i.e. designed for a safety-related application, and therefore fulfills the necessary standards or requirements of the standards or requirements mentioned in the introduction in order to protect a hazardous machine. In particular, it is a camera. The example of a robot is used as the machine in this description, but also includes other machines that pose a risk of accidents, especially for the operating personnel. A human-robot collaboration is specially safeguarded.

The sensor has at least one image sensor with a multiplicity of pixel elements in order to acquire image data of objects in the vicinity of the machine. These image data are processed in a control and evaluation unit in order to detect the objects and at least determine their position. Other parameters that can also be determined, such as the distance between the object and the machine or its speed, and others are also mentioned below. On the basis of the position and, if necessary, further parameters, the danger situation is evaluated and, if necessary, a countermeasure is initiated in order to eliminate a recognized danger. Depending on the situation, this countermeasure can consist of slowing down the movement, evasive action or stopping. The control and evaluation unit can at least partially be implemented externally outside the actual sensor, for example in a safety controller connected to the sensor and / or in a robot controller, and is functionally assigned to the sensor here.

The invention is based on the basic idea of event-based protection. The image sensor is an event-based or neuromorphological image sensor instead of a conventional image sensor. The differences were briefly discussed in the introduction: The pixels each recognize changes in intensity instead of simply measuring the respective intensity. Such a change, which should preferably be quick enough and exceed a noise figure, is one of the eponymous events. In addition, signals are only provided or read out in the event of such a change in intensity, specifically only from the affected pixel or pixels.

The invention has the advantage that event-based protection becomes possible, in particular event-based speed-and-separation monitoring. There is no longer an actual frame rate. The effective repetition rate of the acquisition by the event-responsive pixels is extremely high. These low latencies enable detection and response in practically real time. Only those areas are read out and thus taken into account in which there is a change in intensity. Redundant data such as a static background are not recorded at all. Background segmentation and detection of objects, namely moving objects that are dangerous for protection, is already performed by the event-based image sensor through its operating principle. The image evaluation is significantly simplified and at the same time more robust and thus the sensor is technologically easier to implement and nevertheless improves the monitoring, especially in the case of highly dynamic, close collaboration scenarios. The process flow becomes more efficient, the need for computing power and energy consumption are reduced.

In addition to the very fast pixel trigger rate, an event-based image sensor also enables a high dynamic range. The event-based image sensor is almost insensitive to the constant light component of the background, since this does not cause any change in intensity and therefore does not trigger an event. Most extraneous light scenarios are covered by this, because scene changes are either very sluggish because the light is permanently on, or very briefly when, for example, the light is switched on. Such a brief event quickly disappears in the image data. A modulated light source, such as a halogen lamp modulated at 50 Hz, can be suppressed by the control and evaluation unit. All of these cases show a distinctly different signature than the movement of humans or robots.

It can be seen as an additional advantage that a person can typically be recognized as a person from the image data of an event-based image sensor, but cannot be identified. This is completely sufficient for the security that every person is supposed to protect. In contrast to conventional images, it is no longer necessary to anonymize recognized persons for data protection reasons, because this is already done automatically by the principle of the event-based image sensor.

A respective pixel element preferably determines when the intensity detected by the pixel element changes, and precisely then delivers image information based on an event. The special behavior of the pixel elements of an event-based camera has already been mentioned several times. The pixel element checks whether the recorded intensity changes. Only that is an event, and image information is only output or read out in the event of one event. A type of hysteresis is conceivable in which the pixel element ignores a defined, too slight change in the intensity and does not perceive it as an event.

The pixel element preferably supplies differential information as image information as to whether the intensity has decreased or increased. The information read from the pixel element is therefore, for example, a sign +1 or -1 depending on the direction of change of the intensity. A threshold can be set for intensity changes up to which the pixel element does not yet trigger an event. In a subsequent internal representation for further evaluation, the value 0 can be added for times when no event was recorded.

The pixel element preferably supplies, as image information, an integrated intensity in a time window determined by a change in the intensity. Here the information is not limited to one direction of the change in intensity, but the incident light is integrated in a time window determined by the event and a gray value is thereby determined. The measured value corresponds to that of a conventional camera, but the time of acquisition remains event-based and linked to a change in intensity.

The respective pixel element outputs image information preferably with an update frequency of at least one KHz or even at least ten KHz, one hundred KHz or 1 MHz. This corresponds to latency times from just 1 ms to even just 1 µs. The refresh rate of a traditional camera is the refresh rate or frame rate. The event-based camera does not have such a common image refresh rate, since the pixel elements output or refresh their image information individually and on an event-based basis. However, there are extremely short response times that could only be achieved with a conventional camera at immense costs with thousands or more images per second; with an event-based update frequency that is still possible, this would no longer be technically feasible with conventional cameras. This high temporal resolution also has the advantage that there are practically no motion artifacts (motion blurr). An object does not move several pixels further within one of the extremely short update periods, which corresponds to a conventional exposure time, and therefore no smeared image is recorded.

The image sensor preferably generates a data stream of events as image information, each of which has coordinate information of the associated pixel element, time information and intensity information. A conventional data stream consists of the intensity or gray values of the pixels, and the location reference in the image sensor plane arises from the fact that all pixels are read out in an orderly sequence. In the event-based image sensor, instead, data tuples are preferably output for each event that the event make assignable. The location of the associated pixel element, the direction ± 1 of the change in intensity and / or the intensity measured for the event and a time stamp are recorded.

The data stream is preferably converted into a matrix by filling it with zeros, the basis of which is formed by the arrangement of the pixel elements on the image sensor and the time. For further processing, a familiar matrix display is often more suitable, which consists of a sequence of individual images following one another in time, i.e. of layers of the intensity distribution recorded at a fixed point in time over the pixel elements of the image sensor. The granulation of the time direction would conventionally be specified by the frame rate; with an event-based camera it can be specified much more precisely. The events ultimately generate a sparse matrix, and the data stream from the event-based image sensor corresponds to a memory and bandwidth-saving representation of it.

The event-based image sensor is preferably designed as a secure event-based image sensor through two-channel or self-checking architecture and / or through plausibility checks of the image data by means of the control and evaluation unit. The event-based image sensor that was not developed for safety technology thus fulfills the requirements of the standards mentioned in the introduction. It is conceivable to anchor security directly in the image sensor, but that requires extremely expensive chip development. Instead, two image sensors that are not safe alone can be used, which mutually check each other, or tests and plausibility checks are carried out with only one initially unsafe image sensor at time intervals corresponding to the desired security level. Such tests can be controlled by the image sensor itself or the control and evaluation unit.

The control and evaluation unit is preferably designed to evaluate events detected by the pixel elements as moving objects, in particular signals from pixel elements within safety-relevant areas of interest. If an event is registered in a pixel or pixel area of the event-based image sensor, this is treated directly as a moving object in the evaluation. The object segmentation required with conventional image sensors, including comparison with previous images in order to determine movement, can be dispensed with, since the raw data is already filtered accordingly. However, there may be additional processing or filters. For example, a moving object is ignored that only appears for a very short time and then disappears again and / or that is too small for a part of the body. It is also conceivable to define safety-relevant areas of interest (ROI, region of interest) in the monitoring area and to ignore objects outside as well. Such safety-relevant areas of interest must not be confused with protective fields. Intervention in a protective field would immediately trigger protection of the machine. In contrast, objects are absolutely permitted and even expected in the safety-relevant area of interest; only the area is thereby marked out in which objects are checked for dangerous proximity, speed and the like at all. Outside the safety-relevant areas of interest, there is never a danger due to organizational measures, mechanical barriers or under all conceivable conditions sufficiently large distances.

The control and evaluation unit is preferably designed for object tracking. For this, algorithms based on Kalman filters, for example, are known per se. However, object tracking is considerably simplified because the event-based image sensor already provides very reliable and simple object segmentation for moving objects. The result of an object tracking are, for example, lists with existing objects, previous movement paths of the objects or a forecast for the future movement path including kinematic parameters such as speed and acceleration.

The control and evaluation unit is preferably designed to store the position at which a previously tracked object was last detected as the position of a stationary object and to take it into account in the assessment of a hazard. One consequence of the special way an event-based image sensor works is that stationary objects are not detected. This also applies to a person who stops in the monitored area. It is lost to object tracking, at least temporarily. The control and evaluation unit notes such positions in accordance with this embodiment, and it is thereby known where people are standing, so that this can be taken into account in the protection concept. If the person is outside the range of the robot, it can continue to work at normal speed, because as soon as the person moves again and a reduced working speed is required, this is recognized by the event-based image sensor with the shortest possible latencies. But it is also possible to continue working within the range as long as the robot avoids the known position at which a person is standing. In this case, however, the working speed should preferably be slowed down. Of course, the same applies to the robot that it is only detected when it is in motion, but that is entirely not critical because a stationary robot is not a source of danger.

The control and evaluation unit is preferably designed to search for objects to be newly included in the object tracking in certain partial areas of the monitoring area, in particular at the edge. The specific sub-area is preferably an access, such as a door or an opening in a barrier around a robot cell. In any case, new people and objects can only appear from the edge. When starting up the system, organizational measures should preferably be taken to ensure that nobody is in the vicinity of the robot, because the sensor would only perceive a person standing there if they were moving. This can be solved during operation by object tracking, which also knows the positions of stationary objects, but this information is initially missing. An exemplary solution is that the start-up of the system is reserved for authorized persons, who are then responsible for ensuring that the robot environment is clear.

The control and evaluation unit is preferably designed to define a danger area within the monitoring area which corresponds to possible positions of the machine, in particular by evaluating image data that are recorded during operation of the machine. The hazard area is a special area of interest within the monitored area. If something moves here, in particular an object that the object tracking did not follow from the edge to the robot, this is assigned to the robot. The danger area can be taught-in, for example, by observing the robot for some time during operation by the event-based image sensor without being disturbed by other objects. Precisely those pixels then delivered events that correspond to locations to which a part of the robot has moved at least once during the observation time. The danger area can alternatively be defined from the robot controller or its motion planning, or by hand. A dynamic comparison with the robot controller is also conceivable, in this case a dynamic danger area with variable limits corresponding to the current robot pose is also conceivable.

The control and evaluation unit is preferably designed to check in the image data whether an event has occurred at the same time at a first position within the danger area and at a second position outside the danger area, and in this case a distance between the first position and the second position to be calculated and taken into account when assessing a hazard. Movements within the danger area are assigned to the robot, other movements are not within its range and are therefore to be assigned to a foreign object. The double condition mentioned means that the robot and an object move in its environment at the same time. The environment can remain limited to safety-relevant areas of interest. The distance between the robot and the object is calculated, possibly also other kinematic parameters such as the speed, in particular from the object tracking, and this is evaluated in order to deduce whether the robot must react in a safety-oriented manner. If the double condition does not exist, there are two possible reasons for this. Either nothing moves in the danger area, then the robot stands still and nothing needs to be secured for the moment. Otherwise there is no moving object in the vicinity, which also means there is no imminent danger. It should also be noted that there may well be a stationary object near the robot. The handling of this case by tracking objects and storing positions of objects that are temporarily lost from tracking has already been discussed.

The sensor preferably has a safe output in order to output object positions, distances from objects to the machine and / or additional information derived therefrom, in particular the speed and / or direction of movement. In this case, the further assessment of the danger takes place outside the sensor, which supplies the necessary and, in some cases, already processed detection results. A connected safety controller and / or the robot controller itself can be responsible for this further evaluation. Despite this distribution of functionality, this can still be understood as a control and evaluation unit of the sensor, which then has a double meaning as a component within the sensor and in the broader sense mentioned. The object position will preferably be included in the evaluation, as a rule also the distance. Another important parameter, for example for speed-and-separation monitoring, is the speed, since a nearby, slow object is less endangered than a somewhat more distant, fast object, and similar considerations apply to the direction of movement. Other kinematic parameters such as acceleration can also play a role depending on the embodiment. Object positions are preferably representative, for example an object center of gravity or that object point to which the distance was calculated, and not all known object positions to an object or its object point cloud. However, it is also conceivable to output enveloping bodies for objects or at least the 3D point cloud of the object.

The control and evaluation unit is preferably designed to determine a working speed that eliminates the risk and / or a workflow for the machine that eliminates the risk. If a danger is recognized, for example an object comes too close at too great a speed, an appropriate safety reaction must be determined. The working speed should be as high as safely possible for high productivity. Depending on the embodiment and the situation, the robot remains in its usual trajectory and slows down. Alternatively, a new trajectory is planned with a greater distance to the object, which is also carried out more slowly if necessary. An adapted working speed includes the possibility of a working speed of zero, ie a stop or an emergency stop. All of these adjustments can be made extremely quickly and flexibly thanks to the high effective repetition rate of the event-based image sensor. It has already been stated that the control and evaluation unit can be a component of the sensor in the narrower sense, but can also include the safety controller and / or the robot controller and any functionality distributed over it in the broader sense.

The sensor is preferably arranged on the machine and moved along with its working movements. This applies in particular to the tool tip of a robot arm (EOAS, end-of-arm safeguarding). Other examples are person-guided or autonomous or automatically guided vehicles (AGV, automated guided vehicle).

The control and evaluation unit is preferably designed to identify objects on the basis of a modulated lighting pattern that is generated by a tracker device arranged on the object. The lighting pattern is, for example, a pulsation with an individual, clearly assignable frequency. This causes specific changes in intensity, which the event-based image sensor perceives and which are very easily recognized in the subsequent image analysis. The tracker device can be attached to the robot, for example, so that its movements can be assigned to the robot even without a defined danger area. Other inanimate objects in the vicinity of the robot can also carry a tracker device. Persons to be protected are then very easily recognized by the fact that they are not identified via the tracker device. It is also conceivable, however, that authorized operating personnel also wear a tracker device, in which case even somewhat higher working speeds are permitted on the basis of their particular experience.

The sensor preferably has at least two event-based image sensors in a stereo arrangement, the control and evaluation unit being designed to assign events recorded in the image data of the two image sensors to one another on the basis of which a stereo method is used to calculate a depth value. This creates an event-based stereo camera. This opens up a third dimension of coverage that allows more precise security concepts. In the case of two-dimensional recording, safety margins are made implicitly or explicitly. This leads to unnecessary safeguarding measures, although objectively the three-dimensional distance would not be dangerous at all. While a conventional stereo camera usually searches for correspondences by correlation across all pixels at least parallel to the base distance, in event-based stereo the search for correspondence can be limited to the few pixels that respond. If there is object tracking, in most cases it is already clear with its help, without any pixel-based correlation, which recordings of the two image sensors must belong to one another. In any case, only a few, small pixel areas need to be correlated, since the majority of the pixels in the sparse matrix usually show only zero anyway. The further steps then correspond to the usual stereo method, i.e. the pixel spacing of the corresponding pixels of the two event-based image sensors is converted into a depth value using parameters such as the pixel spacing (pitch) and the basic spacing between the image sensors.

The method according to the invention can be developed in a similar way and shows similar advantages. Such advantageous features are described by way of example, but not conclusively, in the subclaims that follow the independent claims.

• 1 eine schematische dreidimensionale Darstellung einer Kamera und ihres Ü berwachu ngsbereichs;
• 2 eine beispielhafte Intensitätsverteilung eines Pixels zur Erläuterung von ereignisbasierter Erfassung von Bildinformation;
• 3 eine beispielhafte Überwachungssituation aus der Draufsicht einer Kamera gemäß 1 mit einem Roboter und mehreren Personen;
• 4 eine beispielhafte Darstellung der erfassten Bewegungsbahnen eines langsamen und eines schnellen Objekts;
• 5 eine beispielhafte Darstellung der erfassten Bewegungsbahn eines Objekts und einer Prognose für die künftige Bewegungsbahn; und
• 6 eine beispielhafte Darstellung der erfassten Bewegungsbahnen des Roboters und eines Objekts sowie der jeweiligen Abstände dazwischen.

The invention is explained in more detail below also with regard to further features and advantages by way of example using embodiments and with reference to the accompanying drawings. The figures in the drawing show in:

• 1 a schematic three-dimensional representation of a camera and its surveillance area;
• 2 an exemplary intensity distribution of a pixel for explaining event-based acquisition of image information;
• 3rd an exemplary monitoring situation from the top view of a camera according to FIG 1 with a robot and several people;
• 4th an exemplary representation of the detected trajectories of a slow and a fast object;
• 5 an exemplary representation of the detected trajectory of an object and a prognosis for the future trajectory; and
• 6th an exemplary representation of the detected trajectories of the robot and an object as well as the respective distances between them.

1 shows a camera in a schematic three-dimensional representation 10 as an example of an optoelectronic sensor in an arrangement in a bird's eye view for detecting a monitoring area 12th . The entire detectable surveillance area 12th or the work area can be restricted to security-relevant sub-areas via a configuration. The camera 10 is preferably designed to be fail-safe in the sense of safety standards such as those mentioned in the introduction. She has an image sensor 14th with a plurality of pixel elements, in particular arranged in a matrix, which according to the invention is an event-based image sensor 14th which will be discussed in detail later. The image sensor 14th is a lens 16 upstream, which can be implemented in practice as any known imaging lens.

With the image sensor 14th is a control and evaluation unit 18th connected. The control and evaluation unit 18 can be implemented in a wide variety of hardware, for example digital components such as microprocessors, ASICs (Application Specific Integrated Circuit), FPGAs (Field Programmable Gate Array), GPUs (Graphics Processing Unit) or mixed forms thereof. At least parts of the functionality of the control and evaluation unit 18th can be implemented externally, for example in one via a safe output 20th connected safety controller 22nd or a robot controller via the safety controller 22nd or alternatively different from 1 directly connected robot 24 . The robot 24 represents a machine that poses a danger to people 26th goes out in their surroundings.

The control and evaluation unit 18th is with the image sensor 14th connected and evaluates its image data. The aim of this evaluation is to monitor the robot 24 to avoid accidents with people 26th and also to prevent other objects in its vicinity. For this purpose, positions of the robot 24 or its moving parts, positions of people 26th in its environment and depending on the safety concept, additional variables such as distances, speeds and other kinematic or other variables are determined. In the event of danger, the safe output 20th a safety-related reaction initiated. This is only an emergency stop in extreme cases. Thanks to the event-based image sensor 14th is an extremely fast reaction with fine adjustment of the working speed and possibly also the robot's trajectory 24 possible, so that its productivity is restricted as little as possible.

2 shows to explain the functional principle of the event-based image sensor 14th in the upper part a purely exemplary time-dependent intensity profile in a pixel element of the image sensor 14th . A conventional image sensor would integrate this intensity profile over a specified exposure time window, the integrated values of all pixel elements would be output at a specified frame rate and then reset for the next image.

The pixel element of the event-based image sensor 14th instead reacts individually and independently of a frame rate to a change in intensity. Points in time at which a change in intensity was determined are marked with vertical lines. In the lower part of the 2 events at these times are shown with plus and minus depending on the direction of the change in intensity. It is conceivable that the pixel element does not react to any change in intensity, but only when a certain threshold is exceeded. The pixel element is read out event-based at the times at which a change in intensity is detected, and then outputs the sign. The image information can then be supplemented with the value zero at the times when there was no event, so that an image over all pixel elements of the image sensor 14th arises. Unlike conventional images, this picture only shows the moving edges, be it object contours, shadows or imprints.

In addition to such differential event-based cameras, there are also integrating event-based cameras. It reacts analogously to changes in intensity. Instead of outputting the direction of the change in intensity, however, the incident light is integrated in a time window specified by the event. This creates a gray value. Differential and integrating event-based cameras have a different hardware structure, and the differential event-based camera is faster because it does not require an integration time window. For the technology of an event-based camera, reference is also made to the patent literature and scientific literature mentioned in the introduction.

The event-based acquisition of image information creates a sparsely populated (sparse) three-dimensional data set, since unlike a conventional camera, not every pixel element provides image information for a fixed time, but only those that register an event in the form of a change in intensity. An event is, for example, the pixel position on the image sensor as a tuple with the sign of the change in intensity in the case of a differential event-based camera or a gray value in the case of an integrating event-based camera 14th output in X and Y directions and a time stamp. Such events can be added to a normal matrix afterwards by filling them with zeros. Two dimensions then correspond to the pixel arrangement on the image sensor 14th . If time is also taken into account, a three-dimensional matrix is created, the layers of which are each two-dimensional movement images and the third dimension of which is time, preferably with an extremely high time resolution in the range at most a few microseconds or even less. This corresponds to a film with frames formed by the layers, but with an unusually high frame rate.

3rd shows an exemplary monitoring situation in the monitoring area 12th . The security task based on the camera 10 is the presence of people 26th to recognize and depending on their position and possibly other sizes and position and possibly other sizes of the robot 24 initiate a defined reaction in a safety-oriented manner, so that safety is guaranteed at all times. It is again pointed out that robots 24 and people 26th represent any machines or objects in their vicinity with the risk of accidents. The event-based image sensor 14th only recognizes changes in intensity, so that the camera image looks different in reality than in 3rd . Static objects like the base of the robot 24 are not recorded, and neither are the people 26th are not necessarily recognized with all structural features, but only their moving edges.

Due to its functional principle, the event-based image sensor takes 14th only moving objects are true. If a pixel responds, then it is at the corresponding location in the monitoring area 12th an object that must be taken into account in the security concept. However, this does not exclude temporal or spatial filters that, for example, hide an object that only appears very briefly or as a disturbance. However, there is no need for complex segmentation to separate a background and find moving objects; this happens implicitly from the beginning. With a conventional camera, this intermediate result would only be obtained after a large number of steps.

In order not only to evaluate the current situation, but also to obtain other variables such as the speed, the direction of movement or an extrapolation of the movement, the people 26th preferably tracked by an object tracking. The trajectories of movement recorded so far 28 are in 3rd drawn. Areas, not shown, are preferably defined, such as a door or some other access point where object tracking begins. Anyway, become people 26th enter the scene from one of the edges of the picture. The prerequisite for this assumption is that nobody is in the monitoring area when the system is started up 12th stops, because a standing person 26th would not be recorded and could show up somewhere later.

Even if a person previously monitored by object tracking 26th comes to a stop, it is temporarily lost from detection. This position should be the control and evaluation unit 18th note and take into account in the security concept. The robot 24 can continue working, the standing person 26th is not exposed to any danger. Depending on the situation, a maximum working speed can be set depending on the distance between the standing person 26th can be specified. The person begins 26th Moving again is what happens with the extremely low latency of the event-based image sensor 14th recognized and the robot 24 can adjust his working speed practically immediately. The possible working speed is therefore regularly higher than with conventional speed-and-separation monitoring, and this leads to an increase in the efficiency of the process.

The counterpart in the security concept to the people 26th is the robot 24 itself. Its movements are also recorded by the event-based image sensor 14th detected. To them the robot 24 can be assigned to a hazardous area 30th to be defined around the robot. Movements perceived here are then transmitted to the robot 24 assigned. In doing so, exactly the relevant moving part of the robot is also determined 24 recorded with its precise position. The entire danger area is by no means a blanket 30th protected, which would often lead to clearly overestimated intervals and thus unnecessary productivity losses. Elaborate robot models or the like, any knowledge of the robot type or its geometry or communication with the robot controller are not required for this. To the danger area 30th The robot can determine particularly precisely 24 from the image sensor 14th without people 26th can be observed in the vicinity for a certain period of time. Exactly those pixels which, taking into account a noise threshold, may have delivered events during this time belong to the danger zone 30th .

Whenever a moving person at the same time 26th and a moving robot 24 are detected, the mutual distance is calculated depending on the situation and, depending on this, the robot 24 Real time slowed down or accelerated or an evasive movement initiated. Further parameters such as the speed, in particular their component in the direction towards one another, can be taken into account. This distance calculation is much faster than with conventional cameras, as there is no limiting frame rate or the response time of the pixels of the event-based image sensor 14th is much higher. Interestingly, the order of magnitude typically corresponds to the update rate of an event-based image sensor 14th the update rate of path planning commands for a robot 24 . The robot moves 24 not, there is no danger from him and the person 26th may approach at will. Also one out of reach of the robot 24 standing person 26th is not exposed to any danger, although here the working speed of the robot 24 still remains limited depending on the distance. Treatment of dormant people 26th in the context of object tracking has already been discussed.

The robot 24 can be equipped with pulsating lighting in a preferred embodiment. The same applies to known moving objects that are in principle in the vicinity of the robot 24 stop, such as mobile robots or vehicles (AMR, autonomous moblie robots, AGV, automated guided vehicle, AGC, automated guided container). The pulse frequency is set individually so that the robot can be identified 24 or the known object is possible. Instead of a simple modulation frequency, more complex optical codes are also conceivable. There are no aliasing effects here because of the event-based image sensor 14th does not have a fixed frame rate. Moving objects without an identified pulse rate are considered a person 26th treated. Alternatively, authorized persons can 26th also wear pulsating lighting. Such trackers make the evaluation even easier.

4th shows an exemplary representation of the recorded trajectory 28a of a slow object and the trajectory 28b of a fast object. Such trajectories 28a-b are obtained from object tracking, for example. The representation is now different than in 3rd . There, the positions recorded at different points in time were simply lined up in the room or on the floor area. This makes it easy to see where the person is 26th ran, but the time reference is lost. In 4th on the other hand, the XY position can be seen as a function of time. The granulation is because of the extremely fast detection in the event-based image sensor 14th very fine, so that a quasi-continuous time axis results. If two movements differ only in terms of speed, the result is as in 4th a correspondingly compressed or stretched trajectory 28a-b .

bestimmen, wobei dx und dy die Änderung der Pixelkoordinate x beziehungsweise y innerhalb einer Zeit dt ist. Die originären Einheiten sind Pixel beziehungsweise Pixel/Sekunde, aber dies kann bei gemessenem oder unterstelltem Abstand, zum Beispiel indem jedes Objekt vorsichtshalber als auf dem Boden befindlich angesehen wird, in eine übliche Einheit umgerechnet werden.From such time-resolved trajectories 28a-b allows the full vectorial speed including the direction of movement $$v=\left(\begin{array}{c}dx\\ dy\end{array}\right)/dt$$

determine, where dx and dy are the change in pixel coordinates x and y, respectively, within a time dt. The original units are pixels or pixels / second, but this can be converted into a standard unit if the distance is measured or assumed, for example by considering every object to be on the ground as a precaution.

5 shows an exemplary representation of the recorded trajectory 28 of an object and a prognosis or prediction for the future trajectory 32 . For this purpose, the last i determined speeds are considered in order to establish a probability corridor for the movement. The decreasing gray saturation is intended to indicate the decreasing probabilities for a certain position. The future trajectory 32 is shown as a triangle, because the prediction becomes less precise the further one tries to look into the future.

Dabei stehen die Indizes r und o für den Roboter 24 beziehungsweise das Objekt. 6th shows an exemplary representation of the recorded trajectory 28 of an object and the recorded trajectory 34 of the robot 24 as well as the respective distances 36 between. The distances indicated by dashed lines 36 are related to a respective fixed point in time t _i , so that the respective positions at this fixed point in time can be compared with each other in Euclidean terms: $$d=\sqrt{{\left(x_r-x_O\right)}^2+{\left(y_r-y_O\right)}^2}.$$

The indices r and o stand for the robot 24 or the object.

The respective working speed of the robot 24 at time t _i can then be adapted to the object speed v _o and the distance d: v _r (t _i ) = f (v _o (t _i ), d (t _i )). If how to 5 Described the speed is predicted, so can also be used for the future distance between robots 24 and object determine a prediction. The calculation of the working speed of the robot 24 can then also be made dependent on the predictions. Because of the low latencies of the event-based image sensor 14th Adjustments to the working speed are practically possible in real time.

So far, the safeguarding of a robot that can move in itself, but generally remains in one place 24 by a statically mounted camera 10 considered. The camera 10 Alternatively, it is also suitable for mobile applications such as AMR, AGV / AGC, where appropriate safety functions are then defined.

The two-dimensional monitoring can be expanded to a three-dimensional monitoring by using two event-based image sensors 14th operated in a stereo configuration. The stereo correlation does not take place, as is otherwise usual, pixel by pixel over the entire matrix, but only for the few at the same time in both image sensors 14th recorded events, so that the effort is considerably reduced. A conventional segmentation for separating a background, for detecting objects per se and especially only moving objects is not necessary, regardless of the correlation method, since this is what the event-based image sensors do 14th already do it yourself. Monitoring in three dimensions has the advantage that no safety margins have to be made for the calculated distances or conservative assumptions about the distance of the detected objects, such as that they are at the maximum distance. As a result, the protection is adapted even better to the situation, and the robot 24 can work even faster.

Claims (14)

Safe optoelectronic sensor (10) for securing a monitoring area (12) with at least one machine (24), in particular a robot, the sensor (10) having an image sensor (14) with a large number of pixel elements for generating image data of objects (24, 26) in the monitoring area (12) and a control and evaluation unit (18) which is designed to use the image data to detect objects (24, 26) and to at least determine and evaluate their position in order to be able to use the image data to act on the machine (24), characterized in that the image sensor (14) is an event-based image sensor, which by means of two-channel or self-checking architecture and / or by plausibility checks of the image data by means of the control and evaluation unit (18) as a safe event-based image sensor (14 ) is designed, and that the control and evaluation unit is designed to include events detected by the pixel elements as moving objects to treat. Sensor (10) Claim 1 , wherein a respective pixel element determines when the intensity detected by the pixel element changes, and precisely then delivers an event-based image information, wherein in particular the pixel element delivers differential information as image information, whether the intensity has decreased or increased, or an integrated intensity in supplies a time window determined by a change in the intensity and / or wherein the respective pixel element outputs image information with an update frequency of at least one KHz or even at least ten KHz, one hundred KHz or 1 MHz. Sensor (10) according to one of the preceding claims, wherein the control and evaluation unit (18) is designed to evaluate signals from pixel elements within safety-relevant areas of interest as moving objects (24, 26). Sensor (10) according to one of the preceding claims, wherein the control and evaluation unit (18) is designed for object tracking. Sensor (10) Claim 4 , the control and evaluation unit (18) being designed to store the position at which a previously tracked object (26) was last detected as the position of a stationary object (26) and to take it into account in the assessment of a hazard. Sensor (10) Claim 4 or 5 , the control and evaluation unit (18) being designed to search for objects (26) to be newly included in the object tracking in certain sub-areas of the monitoring area (12), in particular at the edge. Sensor (10) according to one of the preceding claims, wherein the control and evaluation unit (18) is designed to define a danger area (30) within the monitoring area (12) which corresponds to possible positions of the machine (24), in particular by evaluation of image data which are recorded during the operation of the machine (24). Sensor (10) Claim 7 , wherein the control and evaluation unit (18) is designed to check in the image data whether an event has occurred at the same time at a first position within the danger area (30) and at a second position outside the danger area (30), and in In this case, to calculate a distance between the first position and the second position and to take it into account when assessing a hazard. Sensor (10) according to one of the preceding claims, which has a safe output (20) to determine object positions, distances from objects (26) to output the machine (24) and / or additional information derived therefrom, in particular the speed and / or direction of movement. Sensor (10) according to one of the preceding claims, wherein the control and evaluation unit (18) is designed to determine a working speed that eliminates the risk and / or a workflow for the machine (24) that eliminates the risk. Sensor (10) according to one of the preceding claims, wherein the sensor (10) is arranged on the machine (24) and moved along with its working movements. Sensor (10) according to one of the preceding claims, wherein the control and evaluation unit (18) is designed to identify objects (24, 26) on the basis of a modulated lighting pattern generated by a tracker device arranged on the object (24, 26) . Sensor (10) according to one of the preceding claims, which has at least two event-based image sensors (14) in a stereo arrangement, and wherein the control and evaluation unit (18) is designed to assign events recorded in the image data of the two image sensors (14) to one another using this to calculate a depth value with a stereo method. Method for securing a surveillance area (12) with at least one machine (24), in particular a robot, an image sensor (14) with a plurality of pixel elements generating image data of objects (24, 26) in the surveillance area (12) and evaluating the image data are to detect objects (24, 26), at least to determine their position and to evaluate in order to recognize a danger and in this case to act on the machine (24), characterized in that the image sensor (14) is an event-based image sensor which is designed as a secure event-based image sensor (14) through two-channel or self-checking architecture and / or through plausibility checks of the image data by means of the control and evaluation unit (18), and that the events recorded by the pixel elements are treated as moving objects.

Images