JP7306840B2 - Object monitoring system with rangefinder - Google Patents

Description

The present invention relates to object monitoring systems with ranging devices, and more particularly to filtering of object monitoring systems.

A TOF (time of flight) camera that outputs a distance based on the time of flight of light is known as a distance measuring device that measures the distance to an object. The TOF camera uses a phase difference method that irradiates the target space with reference light whose intensity is modulated at a predetermined cycle, and outputs the distance measurement value of the target space based on the phase difference between the reference light and the reflected light from the target space. There are many things to do. The following documents are well known as object detection techniques that also assume the use of TOF cameras.

In Patent Document 1, a background difference image is generated between a captured image of a detection area captured by a TOF camera or the like and a background image of the detection area, and the foreground pixels are grouped in the background difference image. It is described that a foreground area having a number of pixels equal to or less than the determination pixel count is determined to be noise.

JP 2014-56494 A

When judging the presence or absence of an object within the monitoring area defined in the range space of the TOF camera, the distance measurement value of the TOF camera varies. Pixels that are determined to be within the monitored area may be discrete. TOF cameras may also produce pixels that exhibit ranging anomalies such as saturation, underexposure, image sensor pixel failure, and the like. When determining the presence or absence of an object in a monitoring area using a rangefinder that generates such variations in range values and range measurement abnormalities, it is necessary to perform filter processing on the range image in order to stabilize the determination. is assumed.

However, with conventional image processing filters such as contraction filters, averaging filters, median filters, etc., if the number of pixels determined to be within the monitoring area is small relative to the filter size, it cannot be determined that there is an object within the monitoring area. There is For example, the erosion filter outputs 1 when all 3×3 pixel values around the target pixel are 1, as shown in FIG. 11A. The general effect expected of an erosion filter is to remove noise and fine irregularities that are smaller than a specified size. If all the distance measurement values of the pixel group are not within the monitoring area, it cannot be determined that an object has entered the monitoring area.

Also, the averaging filter outputs the average value of 3×3 pixel values around the target pixel as shown in FIG. 11B. The general effect expected with an averaging filter is to smooth areas that change sharply. If the distance is averaged, it cannot be determined that an object has entered the monitoring area.

Furthermore, the median filter outputs the median value (for example, the fifth value) of the 3×3 pixel values around the target pixel, as shown in FIG. 11C. The general effect expected with a median filter is the removal of spike-like noise, and while the smoothness is slightly inferior to that of an averaging filter, blurring of edges is suppressed. When detecting an object within a monitoring area by applying a median filter to a range image, if five or more pixels in a 3×3 pixel group are not determined to be within the monitoring area, the object has entered the monitoring area. cannot be determined.

Therefore, in an object monitoring system using a distance measuring device that causes variations in measured distance values, abnormal distance measurement, etc., there is a demand for filter processing that enhances the stability of object detection.

One aspect of the present disclosure is an object monitoring system that includes a distance measuring device that generates a range image of a target space, and a computer that determines the presence or absence of an object within a monitoring area defined in the target space based on the range image. In the distance image, the computer device determines that the number of pixels determined to be within the monitoring area among the first number of adjacent pixel groups having a specific arrangement relationship in the range image is at least one of saturation, underexposure, pixel failure, and accuracy abnormality. a filter that determines that there is an object in the monitoring area when the number of pixels including the number of pixels indicating a ranging abnormality is determined to be equal to or greater than a second number, which is equal to or greater than 1 and less than a first number, An object monitoring system is provided.

According to one aspect of the present disclosure, object detection becomes possible even when object detection is difficult with a conventional image processing filter. Moreover, even if there is a pixel with an abnormality in ranging, determination can be continued only with the filter of the present disclosure.

1 is a block diagram of an object monitoring system in one embodiment; FIG. It is a figure which shows the state of the distance image explaining the specific example of filter processing. It is a figure explaining the modification of a filter process. It is a figure explaining other modifications of filter processing. It is a figure which shows an example of a filter shape. It is a figure explaining the selection algorithm of the judgment reason of a filter. 10 is a drawing-substituting photograph showing an example of an image in which pixel states are superimposed on an image of a target space; 4 is a histogram showing variations in distance measurement values; 10 is a graph showing normal distribution and cumulative distribution probability F(x) of distance measurement values having an average μ and deviation σ; The relationship between the decision threshold x and the decision probability of the filter of the present disclosure is added for each N. FIG. 2 illustrates a conventional erosion filter; FIG. 10 is a diagram showing a conventional averaging filter; FIG. 2 illustrates a conventional median filter;

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In each drawing, the same or similar components are given the same or similar reference numerals. Moreover, the embodiments described below do not limit the technical scope of the invention described in the claims and the meaning of the terms.

FIG. 1 is a block diagram of an object monitoring system 1 according to this embodiment. The object monitoring system 1 includes a distance measuring device 10 and a computer device 20 . The distance measuring device 10 and the computer device 20 may be connected so as to be able to communicate with each other via a wired or wireless network, or may be integrally configured by a bus connection or the like.

The rangefinder 10 is composed of a TOF camera, a laser scanner, etc., and sequentially generates range images of a target space. The distance measuring device 10 includes an input/output unit 11 , a light emission imaging control unit 12 , an irradiation unit 13 , a light receiving unit 14 , an A/D conversion unit 15 and a distance image generation unit 16 . The input/output unit 11 inputs various setting values and outputs distance images and intensity images. The light emission imaging control unit 12 controls the light emission of the irradiation unit 13 and the imaging of the light receiving unit 14 based on the input set value.

The irradiation unit 13 includes a light source that emits intensity-modulated reference light, a scanner mechanism such as a diffuser plate and a MEMS mirror for irradiating the target space with the reference light, and the like. The light receiving unit 14 includes a condenser lens that collects the reflected light from the target space, an optical filter that transmits only the reflected light of a specific wavelength, a light receiving element that receives the reflected light, and the like. The light-receiving unit 14 repeats light reception at four types of exposure timings that are phase-shifted by, for example, 0°, 90°, 180°, and 270° with respect to the emission timing of the reference light, and charges Q ₁ -Q ₄ for each phase. accumulate.

The A/D converter 15 amplifies the accumulated charge amounts Q ₁ -Q ₄ , A/D-converts them, and outputs them as binarized values. If saturation is detected during amplification, a value (bit) indicating this is output.

The distance image generation unit 16 obtains the phase difference between the reference light and the reflected light for each pixel based on the charge amount Q ₁ -Q ₄ , calculates the distance measurement value from the phase difference of each pixel, and generates a distance image. . An example of calculation formulas for the phase difference Td and the distance measurement value L is shown below. In the following equation, c is the speed of light and f is the modulation frequency of the reference light.

Further, based on the charge amount Q ₁ -Q ₄ , the distance image generation unit 16 generally determines that saturation occurs in the output of the A/D conversion unit 15 when saturation of the charge amount is detected, and sets the distance measurement value to A singular value (eg, 9999) can also be output. Further, when it is determined that all of the charge amounts Q ₁ -Q ₄ are smaller than the specified value, the distance image generation unit 16 determines that the exposure is insufficient, and outputs a singular value (for example, 9998) as the distance measurement value. You can also

The distance image generator 16 can also determine that a pixel failure has occurred and output a singular value (for example, 9997) as a distance measurement value when all of the charge amounts Q ₁ -Q ₄ are the same value. If the accuracy information obtained by adjusting the scale of the difference between the sum of the charges Q ₁ and Q ₃ and the sum of the charges Q ₂ and Q ₄ exceeds a predetermined threshold, it is determined that the accuracy is abnormal, and the singular value (eg 9996) can also be output. See Japanese Patent Application No. 2018-112665 for details of the accuracy information. Incidentally, in another embodiment, regarding the detection of these abnormalities and the operation of replacing with singular values, the A/D conversion values of the amounts of charge Q ₁ -Q ₄ are sent to the computer device 20, and the computer device 20 makes a decision. may

The rangefinder 10 may further include an intensity image generator 17 . The intensity image generation unit 17 calculates the received light intensity I based on the relationship between the charge amounts Q ₁ -Q ₄ for each pixel in the distance image, and sequentially generates intensity images corresponding to the distance image. An example of the formula for calculating the received light intensity I is shown below.

The computer device 20 is composed of a CPU (central processing unit), a RAM (random access memory), an ASIC (application specific integrated circuit), an FPGA (field-programmable gate array), and the like. based on the distance image. The computer device 20 includes an input/output unit 21 , a setting memory 22 , a filter determination unit 23 , a signal output unit 24 and a display unit 25 .

The input/output unit 21 inputs distance images, intensity images, etc., and outputs various set values. The setting memory 22 stores monitoring area data, various filter setting values, imaging modes, and the like. The monitoring area data is set as a three-dimensional position in the target space of the distance measuring device 10, converted into a distance range table of the monitoring area viewed by each pixel of the distance image, and stored. The filter setting value is a setting value related to a filter for determining the presence or absence of an object within the monitoring area, and includes, for example, a filter shape, a determination threshold value, and other setting values described later. The imaging mode is a setting value related to luminous imaging of the distance measuring device, and includes, for example, the amount of reference light emitted, the exposure time, the aperture value, and the like.

A filter determination unit 23 filters the distance image to determine the presence or absence of an object within the monitoring area. The filter of this example has N or more (N is an integer), it is determined that there is an object in the monitoring area.

A signal output unit 24 outputs an object detection signal 26 to the outside when it is filtered that there is an object in the monitoring area. The object detection signal 26 is used, for example, as a power stop signal for dangerous sources such as robots and machine tools that are isolated from workers in order to ensure the safety of workers who have entered the monitored area.

The display unit 26 displays various setting screens, distance images, intensity images, and the like. As will be described later, the display unit 26 of this example superimposes the pixel status such as whether or not the object is within the monitoring area, the type of range finding abnormality, and the filter determination result on the image of the target space such as the range image and the intensity image. Display an image.

FIG. 2 is a diagram showing the state of a distance image for explaining a specific example of filtering. A pixel area 31 indicates pixels that overlook the set monitoring area, and if the distance measurement values of these pixels are within the range of the distance range table, they are determined to be within the monitoring area. In FIG. 2, it is assumed that part of the object 30 has entered the monitoring area. Note that the distance image 32 includes four pixels 32a to 32d determined to be within the monitoring area within the pixel area 31, and one pixel 32e indicating an abnormality in distance measurement. Note that FIG. 2 shows only a portion of range image 32 and that object 30 is represented on range image 32 for ease of understanding.

The filter 33 of this example selects the monitor region 31 when it is determined that the number of pixels determined to have entered the monitor region 31 is N=4 or more among the 3×3=M=9 adjacent pixel groups. 31 determines that there is an object. Generally, in filtering, a target pixel is positioned at the center of a group of adjacent pixels, and a determination is made using the values of a group of adjacent pixels including the target pixel. When the filter 33 is applied to the range image 32 and there is even one pixel of interest determined to contain an object within the monitoring area 31, an object detection signal is output. For example, as shown in FIG. 2, when four target pixels 32a to 32d are set as pixels of interest, the number of pixels determined to be within the monitoring region 31 is four or more. (The filter determination result is shown on the right side of FIG. 2.). Even if there is a pixel 32e indicating a range finding abnormality in the range image 32, a similar filter determination result can be obtained, so the filter 33 of this example alone can be used to continue the determination.

In contrast, in conventional image processing filters, such as erosion filters and median filters, the number of pixels determined to be within the monitoring region 31 is not 9 or 5, so there are actually no objects within the monitoring region 31 . Even if 30 is intruding, the object cannot be detected. Also in the case of the averaging filter, since the distance measurement values of the surrounding pixels other than the target pixels 32a to 32d are dominant in calculating the average value, there is a high possibility that the object cannot be detected. Therefore, according to the filter 33 of this example, it is possible to detect an object even when it is difficult to detect an object with a conventional image processing filter.

FIG. 3 is a diagram for explaining a modified example of filtering. In FIG. 3, it is assumed that the retroreflective material 40 is attached to part of the tip of the object 30 . Since saturation occurs due to the retroreflection material 40, the range image 32 includes six pixels 40a-40f indicating abnormal range measurement in addition to the three pixels 32a-32c determined to be within the monitoring area 31. FIG.

The filter 33 of this example determines whether or not the number of pixels including the number of pixels indicating a ranging abnormality such as saturation is N=4 or more in the number of pixels determined to be within the monitoring area 31. is different. When the three pixels 32a to 32c determined to be within the monitoring area 31 and the six pixels 40a to 40f indicating the range finding abnormality are set as the pixels of interest, all of them are determined to be within the monitoring area 31. Since the number of pixels including the number of pixels indicating distance measurement abnormality such as saturation is four or more, it is determined that there is an object within the monitoring area 31 (filter determination results are shown on the right side of FIG. 3). ).

In contrast, with the filter that does not include the number of pixels with abnormal ranging, the number of pixels determined to be within the monitoring area 31 is only three. Cannot be judged as present. In this way, by including the number of pixels indicating a range finding error in the filter judgment, it is possible to judge objects with retroreflective material on the tip as shown in this example more quickly and on the safer side. be possible.

FIG. 8 shows variations in distance measurement values output by pixels viewing an object about 2 m away. It is known that the dispersion of the distance measurement value of each pixel of the TOF camera is distributed like a normal distribution because it is affected by unavoidable random noise such as shot noise, dark current noise, and thermal noise. This means that when an object is exactly on the far side of the monitored area, the determination probability of determining that each pixel viewing the object is within the monitored area is approximately 0.5. FIG. 9 shows a normal distribution of measured distance values having an average μ and deviation σ and its cumulative distribution probability F(x). When μ is the distance in the far plane of the monitoring area, the cumulative distribution probability F(x) corresponding to the determination probability is 0.5. The cumulative distribution probability F(x) corresponding to the probability increases.

FIG. 4 is a diagram for explaining another modified example of filtering. As shown in FIG. 9, if the measured distance values have a deviation of σ and an object is intruding from the far side of the monitoring area by a distance corresponding to 0.25σ, each pixel viewing the object is monitored. The determination probability of determination as being in the area is 0.6. Accordingly, in each distance image, the states of pixels determined to be within the monitoring area are discrete as shown in FIG. Consider the filter decision probability after filtering when 3×3=M=9 and N=4 filters are applied under this situation. Let p be the determination probability that each pixel viewing an object is determined to be within the monitoring area, and let P be the filter determination probability after filtering. The filter determination probability P is 0.733, which is obtained from the following binomial distribution formula for calculating the probability of success at least once.

<Method 1>
In order to increase the filter determination probability, the filter 33 of method 1 indicates the sum of the number of pixels determined to be within the monitoring area over the Q distance images n, n-1, and n-2 generated in time series, or the distance measurement abnormality. This is different from the previous one in that it is a temporal filter that determines whether the total number of pixels including the number of pixels is L or more, which is 1 or more and less than M×Q. For example, as shown in FIG. 4, 3×3 M=9×3=27, N=4× When 3=12 temporal filters are applied, the filter decision probability P1 improves to 0.957. Therefore, the stability of object detection is enhanced.

<Method 2>
In order to further increase the filter determination probability, the filter 33 of method 2 determines the number of pixels determined to be within the monitoring area even once over the Q distance images n, n−1, and n−2 generated in time series, or This is different from the above-described method in that it is determined whether the number of pixels including the number of pixels indicating is N or more. For example, as shown in FIG. 4, when 3×3 M=9 and N=4 temporal filters are applied over the past three distance images n, n−1, and n−2, the determination probability of 1 pixel is Since p improves from 0.6 to 0.936, the filter decision probability P2 improves to 0.9999. Therefore, the stability of object detection is further enhanced.

The degree of improvement in determination probability increases as the number of distance images to be filtered increases. Incidentally, when the time filter constant Q=10, the filter decision probability P1 improves to 0.997, and the filter decision probability P2 improves to 1-1.6E-18.

Depending on the aforementioned filters, the computing device 20 may comprise means for setting at least one of the following items. The setting means includes setting software, setting buttons, and the like.
(1) Filter shape (filter size: M, layout relationship of pixel groups, etc. An example is shown in FIG. 5)
(2) Filter decision threshold: N, L
(3) Whether or not to include the number of pixels indicating a ranging error (4) Time filter constant: Q
(5) Selection of filter to be used (6) Imaging mode (emission amount of reference light, exposure time, aperture value, etc.)
By setting the above items, it is possible to adjust the object detection size, filter determination probability, etc., so flexible object detection according to the purpose of use, installation environment, etc. is possible.

Further, when the computer device 20 performs filter determination including the number of pixels indicating a ranging abnormality, the reasons for determining that there is an object in the monitoring area are the intrusion of the object and the ranging abnormality. It may be classified into a plurality of categories such as saturation, underexposure, pixel failure, accuracy abnormality , etc., and it is preferable to provide means for notifying the reason for determining that an object exists within the monitoring area. Notification means include a notification mail, a notification sound, a notification lamp, and the like.

FIG. 6 is a diagram for explaining an algorithm for selecting reasons for filter determination. As an algorithm for selecting the reason for determination in the filtering process, there is a method of selecting the most frequently seen type in the filter. The types include intrusion, saturation, underexposure, pixel failure, and precision abnormality. In FIG. 6, it is assumed that the range image includes three pixels 32a-32c determined to be within the monitored area, two pixels 50a-50b indicating saturation, and one pixel 50c indicating underexposure. are doing. When it is determined that there is an object in the monitoring area by applying the filter 33 that includes the number of pixels indicating a ranging abnormality to the target pixel 32b, the most common type in the filter is "intrusion", so the reason for the determination of the filter 33 is selected as "invasion". In addition, there is a method of selecting a reason with a high weighted average that takes into consideration the weight that decreases as the distance from the target pixel increases, and a method of assigning priorities to the determination reasons and selecting the reason with the highest priority.

On the other hand, the reason for determining that there is an object in the monitoring area of the object monitoring system is generally the reason for all the filter determinations of the same range image in which the filter determination determined that there is an object. Therefore, there are cases where there are multiple reasons for filter determination in each filter process, and there are cases where the types of the reasons are different.

Further, when there are a plurality of different reasons for filter determination for each pixel of interest, it may be easier for the user to select and notify the main reason. As an algorithm for selecting the main reason for determining that there is an object in the monitored area as an object monitoring system, there is a method of selecting the most common reason for determination, or a method of prioritizing the reasons for determination and determining the reason for determination with the highest priority. There is a method of selecting . As an example, by setting "intrusion" to the highest priority, for example, an object with a retroreflective material entered, three filters judged saturation, and one filter judged object intrusion. In this case as well, "intrusion" is notified as the main reason for determining that there is an object in the monitored area, allowing the user to select a more realistic reason.

Further, the computer device 20 determines whether or not it is within the monitored area (non-intrusion, intrusion), the type of abnormality in distance measurement (saturation, insufficient exposure, pixel failure, accuracy abnormality, etc.), the filter determination result, and the filter determination reason. Means may be provided for displaying at least one pixel state overlaid on the image of the object space. Display means includes a liquid crystal display and the like. FIG. 7 shows an image of the pixel states overlaid on the intensity image 34 of the object space. Such a superimposed image of pixel conditions can be used to effectively check the initial image condition when the object monitoring system is installed, the effect of improvement measures, the test of object intrusion detection into the monitored area, etc. Furthermore, it is useful for investigating the cause when an object is detected.

The filter of the present disclosure has an effect equivalent to reducing variations in distance measurement values. The range of variation in the measured distance values distributed in the normal distribution shown in FIG. 8 can be statistically represented by the deviation σ. Needless to say, the smaller the deviation σ, the better for the distance measuring device. FIG. 9 shows a normal distribution of measured distance values having an average μ and deviation σ and its cumulative distribution probability F(x). Assuming that the distance in the far plane of the monitoring area is μ, when an object is exactly on the far side of the monitoring area and μ is applied to the judgment threshold value x for the distance measurement value of the pixels viewing this, it is judged as μ or less. This indicates that the probability of determination to be made is 0.5. Similarly, when μ-σ is applied to the determination threshold x, the determination probability of being determined to be μ-σ or less is 0.16, and when μ+σ is applied to the determination threshold x, the determination is determined to be μ+σ or less. The probability is 0.84. That is, the range of determination probabilities from 0.16 to 0.84 is 2σ.

In considering the effect of reducing the distance measurement variation by the filter of the present disclosure, as an example, M = 9 filters, nine target pixels measure the distance of an object at the same distance, and the same variation distribution characteristic: cumulative distribution probability F (x) and determined with the same determination threshold value x. At this time, in each case of N=1 to N=8, the determination threshold value x to be applied and the filter determination probability P at that time are obtained by the following binomial distribution formula.

FIG. 10 adds the relationship between the decision threshold x and the decision probability of the filter of the present disclosure for each N. In FIG. From the graph of the cumulative distribution probability F(x) of individual pixels, it can be seen that the rising edge of the filter decision probability for N=1 to N=7 becomes steep. Furthermore, the following table summarizes the above-mentioned determination probability x of 0.16, the determination probability of 0.84, and the determination probability of 0.5 for each graph. In the table, the difference between the determination probabilities of 0.16 and 0.84 and the ratio of the deviation σ compared with a single pixel are added as steepness indices.

For any N, the width of the determination threshold at which the determination probability becomes 0.16 to 0.84 is around 1σ, and the filter of the present disclosure reduces the variation in the distance measurement value of a single pixel to about half. It can be seen that it has an effect equivalent to

The components of the computer device 20 described above may be configured as programs executed by a CPU or the like. Such a program can be provided by being recorded on a computer-readable non-transitory recording medium such as a CD-ROM.

Although various embodiments have been described herein, it is recognized that the present invention is not limited to the embodiments described above and that various modifications can be made within the scope of the following claims. want to be

Reference Signs List 1 object monitoring system 10 rangefinder 11 input/output unit 12 light emission imaging control unit 13 irradiation unit 14 light receiving unit 15 A/D conversion unit 16 distance image generation unit 17 intensity image generation unit 20 computer device 21 input/output unit 22 setting memory 23 Filter determination unit 24 Signal output unit 25 Display unit 26 Object detection signal 30 Object 31 Monitoring area 32 Distance images 32a-32d Pixels determined to be within the monitoring area 32e Pixels indicating distance measurement abnormality 33 Filter 34 Intensity image 40 Retroreflection material 40a- 40f Pixels indicating ranging abnormality 50a-50b Pixels indicating saturation 50c Pixels indicating underexposure

Claims (9)

An object monitoring system comprising: a distance measuring device that generates a range image of a target space; and a computer that determines the presence or absence of an object within a monitoring area defined in the target space based on the range image,
In the distance image, the computer device determines that the number of pixels determined to be within the monitoring area, among the first number of adjacent pixel groups having a specific arrangement relationship , is at least one of saturation, underexposure, pixel failure, and accuracy abnormality. A filter for determining that there is an object in the monitoring area when the number of pixels including the number of pixels indicating a ranging abnormality is determined to be equal to or greater than a second number that is equal to or greater than 1 and less than the first number. An object monitoring system, comprising: 2. The object monitoring system according to claim 1, wherein said computer device further comprises means for setting at least one of said first number, said second number, and said arrangement relationship. The computer device determines that the number of pixels determined to be within the monitoring area among the adjacent pixel group over the third number of the range images generated in time series has at least one of saturation, underexposure, pixel failure, and accuracy abnormality. Further comprising a filter for determining whether the total number of pixels including the number of pixels indicating a certain ranging abnormality is equal to or greater than 1 and is equal to or greater than a fourth number, which is less than the number obtained by multiplying the first number by the third number. 3. An object monitoring system according to item 1 or 2 . 4. The object monitoring system of claim 3 , wherein said computing device further comprises means for setting said fourth number. The computer device determines that the number of pixels determined to be within the monitoring area among the adjacent pixel group even once over the third number of distance images generated in chronological order has at least saturation, underexposure, pixel failure, and accuracy abnormality. 5. The object monitoring system according to any one of claims 1 to 4 , further comprising a filter for determining whether the number of pixels including the number of pixels indicating one ranging abnormality is equal to or greater than the second number. 6. The object monitoring system according to any one of claims 3 to 5 , wherein said computer device further comprises means for setting said third number. 7. An object monitoring system according to any one of claims 1 to 6 , wherein said computing device further comprises means for selecting filtering to be used. 8. The object monitoring system according to any one of claims 1 to 7 , wherein said computer device further comprises means for notifying a reason for determining that an object exists within said monitoring area. The computer device determines whether or not the area is within the monitoring area, a type of ranging abnormality that is at least one of saturation, underexposure, pixel failure, and accuracy abnormality, a determination result of the filter, and a reason for the filter determination. 9. An object monitoring system according to any preceding claim, further comprising means for displaying one pixel situation on the image of the object space.

Images