WO2020175085A1

WO2020175085A1 - Image processing apparatus and image processing method

Info

Publication number: WO2020175085A1
Application number: PCT/JP2020/004719
Authority: WO
Inventors: 俊之村松
Original assignee: コニカミノルタ株式会社
Priority date: 2019-02-26
Filing date: 2020-02-07
Publication date: 2020-09-03
Also published as: JP7375806B2; JPWO2020175085A1

Abstract

This image processing apparatus is provided with: a first analysis unit (30) which, on the basis of range images arranged in time series, generates first tracking information (D1) obtained by tracking the positions of a moving body present in a prescribed region; a second analysis unit (40) which, on the basis of camera-captured images arranged in time series, generates second tracking information (D2) obtained by tracking the positions of a moving body present in the prescribed region; and a data comparison processing unit (50) which makes a comparison between the first tracking information (D1) and the second tracking information (D2) with respective temporal axes thereof being aligned with each other and then determines the position and identification information of the moving body present in the prescribed region on the basis of the first tracking information (D1) and the second tracking information (D2).

Description

Specification

Title of invention: Image processing apparatus and image processing method

Technical field

The present disclosure relates to an image processing device and an image processing method.

Background technology

[0002] There is known an image processing device that detects the position of an object existing in an imaging region based on an image captured by a camera or the like. This type of image processing device accurately grasps the position of a person or a working machine (hereinafter collectively referred to as “moving body”) in a work environment such as a construction site or a factory and analyzes the action or motion of the moving body. It is also expected to be applied to applications for predicting the behavior or motion of the moving body (hereinafter collectively referred to as “behavior analysis”).

[0003] There are various technologies for recognizing the position of a moving body, but especially in an environment where dust and electromagnetic waves are reverberant, such as a construction site, laser radar (Lig ht Detect i on And Rang i ng Position measurement using light reflection such as: (also called L i DAR) is effective.

[0004] FIG. 1 is a diagram showing an example of a range image generated by a laser radar.

[0005] In general, a laser radar emits a laser beam and measures the time (T0F: Time of Fli ght) until the laser beam is reflected by an object and returns to the object from its own position. Find the distance to the position of the object. Then, the laser radar 1 performs the processing while scanning within a predetermined range in which the monitoring target area is reflected, thereby generating image data related to the range image. Since such a range image includes information on the three-dimensional position of each part of the moving body, it is useful for recognizing the posture and motion of the moving body.

[0006] In Patent Document 1, the surrounding environment is measured using a laser radar, and the moving object is recognized by clustering and analyzing the range-finding points that have been made into a point cloud, thereby recognizing the position of the moving object. The technique to be tried is disclosed.

Prior art documents \¥0 2020/175085 2 (: 17 2020/004719 Patent Document

[0007] Patent Document 1: Japanese Patent Laid-Open No. 20 1 4 _ 1 6 7 7 02

Problems to be Solved by the Invention

By the way, in this type of image processing apparatus, in order to analyze the behavior of a moving body, it is necessary to continue to recognize the same object in a plurality of images (frames) arranged in time series. That is, in this type of image processing apparatus, there is a demand for improving the accuracy of tracking of a moving object shown in a moving image.

[0009] In this respect, Patent Document 1 limits the improvement of the accuracy of the object recognition because the object recognition is performed using only the measurement result of the laser radar. In particular, the range image generated by the laser radar has lower horizontal and vertical resolution than the camera image generated by a general camera, and in addition, Does not include color information. Therefore, for example, if there are multiple people with similar physical characteristics in the area to be monitored by the laser radar, the distance images generated by the laser radar should be used to correctly track the multiple people. Is difficult.

[0010] The present disclosure has been made in view of the above problems, and an object of the present disclosure is to provide an image processing device and an image processing method capable of improving the accuracy of moving object tracking using a laser radar. Means for solving the problem

[001 1] The main disclosure for solving the above-mentioned problems is

A first image acquisition unit that acquires range image data from a laser radar that monitors a predetermined area;

A second image acquisition unit that acquires image data of a camera image from a camera that monitors the predetermined area,

A first analysis unit that generates first tracking information that tracks the position of a moving body existing in the predetermined region based on the distance images arranged in time series, 〇 2020/175085 3 (:171? 2020/004719

A second analysis unit that generates second tracking information that tracks the position of a moving object existing in the predetermined area based on the camera images arranged in time series, and the first tracking information and the second tracking information with time. A data comparison processing unit that aligns and compares the axes, and determines the identification information and the position of the moving object existing in the predetermined area based on the first tracking information and the second tracking information,

An image processing apparatus including.

[0012] Also, in another aspect,

Image data of a range image is acquired from a laser radar that monitors a predetermined area.

Image data of a camera image is acquired from a camera that monitors the predetermined area, and based on the distance images arranged in time series, first tracking information that tracks the position of a moving body existing in the predetermined area is generated,

Based on the camera images arranged in time series, second tracking information is generated by tracking the position of the moving body existing in the predetermined area,

The first tracking information and the second tracking information are compared with each other on the same time axis, and based on the first tracking information and the second tracking information, the identification information and the position of the moving object existing in the predetermined area are determined. Determine,

This is an image processing method.

Effect of the invention

[0013] According to the image processing device of the present disclosure, it is possible to improve the accuracy of moving body tracking using a laser radar.

Brief description of the drawings

[0014] [FIG. 1] A diagram showing an example of a range image generated by a laser radar.

[Fig. 2] Diagram showing an example of the monitoring system according to the first embodiment.

[FIG. 3] A diagram showing a hardware configuration of the image processing apparatus according to the first embodiment [FIG. 4] A diagram showing functional blocks of the image processing apparatus according to the first embodiment [FIG. The figure which shows an example of the tracking information generated 〇 2020/175085 4 卩 (：171? 2020 /004719

[Fig.58] A diagram showing an example of tracking information generated by the second analysis unit.

[Fig. 6] Diagram for schematically explaining the processing of the data comparison processing unit

[Figure 7] View of the field of view of the laser radar and camera

[FIG. 8] A flow chart showing processing executed when the image processing apparatus (data comparison processing unit) according to the first embodiment generates fixed tracking information.

[FIG. 9] A diagram showing a configuration of an image processing apparatus according to a second embodiment.

[FIG. 10] A diagram showing a configuration of an image processing apparatus according to a third embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0015] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, constituent elements having substantially the same function are designated by the same reference numeral, and repeated description will be omitted.

[0016] (First embodiment)

[Overall configuration of the monitoring system]

Hereinafter, with reference to FIG. 2 to FIG. 4, an outline of the configuration of the surveillance system according to the embodiment and the configuration of the image processing apparatus applied to the surveillance system will be described.

[0017] FIG. 2 is a diagram showing an example of the monitoring system II according to the present embodiment. The monitoring system according to the present embodiment is a moving object (here, a person IV!

Work machine

It is used for tracking applications.

[0018] A monitoring system II according to this embodiment includes an image processing device 100, a laser radar 200, and a camera 300.

[0019] The laser radar 200 is, for example, a time until the laser light is emitted and the laser light is reflected by an object and returns.

○ † 4 4) is measured to obtain the distance from the position of the body to the position of the object. The laser radar 200 performs such processing while scanning within a predetermined range in which the area to be monitored is imaged, thereby generating image data related to the distance image (hereinafter, abbreviated as “distance image”). The laser radar 200 continuously generates distance images in frame units and outputs distance images arranged in time series (that is, moving images) to the image processing device 100. 〇 2020/175085 5 (:171? 2020/004719

[0020] The range image has a laser radar 20 for each pixel with each scanning position as a pixel.

It is an image in which measurement data of 0 (for example, distance and reflection intensity) are associated as pixel values (also referred to as point cloud data). The range image shows a three-dimensional (eg, horizontal, vertical, and depth) position of the object in the monitored area. For example, the position of the object is detected in a three-dimensional Cartesian coordinate system (X , 丫, ).

[0021] The camera 300 is, for example, a general visible camera, which converts an image signal generated by its own image pickup device (0 1\/1 0 3 sensor or 0x00 sensor) into 80 , Image data (hereinafter referred to as “camera image”) is generated. Note that the camera 300 continuously generates camera images in frame units and outputs camera images arranged in time series (that is, moving images) to the image processing apparatus 100.

[0022] The camera image is, for example, for each pixel,

Luminance value for each (for example,

◦ It is an image in which the brightness values of 2 5 6 gradations for each of the Tami) are associated as pixel values.

[0023] The laser radar 200 and the camera 300 are installed at appropriate positions near the monitoring target area so that the same monitoring target area is imaged.

[0024] The image processing device 100, based on the image data of the range image generated by the laser radar 200 and the image data of the camera image generated by the camera 300, Existing in (in Fig. 2,

Work machine

It tracks the movement of the and outputs the tracking result.

FIG. 3 is a diagram showing a hardware configuration of the image processing apparatus 100 according to the present embodiment.

[0026] The image processing apparatus 100 has, as a main component,

It is a computer equipped with 0111111800㊀1^111〇 ") 103, an external storage device (for example, flash memory) 104, and a communication interface 105.

[0027] In each function of the image processing apparatus 100 described later, for example, 091\]01 is

Control programs stored in 1 0 2, [¾ 8 1\/1 10 3, external storage device 10 4 etc. It is realized by referring to (for example, an image processing program) and various data. However, a part or all of each function may be realized by processing by a DSP (Digital Signal Processor) instead of or together with the processing by the CPU. Similarly, part or all of each function may be realized by a process by a dedicated hardware circuit (for example, ASIC or F PGA) instead of or together with the process by software.

[0028] [Configuration of image processing apparatus]

FIG. 4 is a diagram showing functional blocks of the image processing apparatus 100 according to this embodiment. The arrows in Fig. 4 represent the flow of data.

FIG. 5A is a diagram showing an example of tracking information D 1 (hereinafter, referred to as “first tracking information”) generated by the first analysis unit 30, and FIG. FIG. 6 is a diagram showing an example of tracking information D 2 (hereinafter, referred to as “second tracking information”) generated by 40. 5A and 5A show the position of the moving body in each claim, and D (corresponding to "identification information" of the present invention) represents the identification number of each moving body, and t = 0, t = 1 and t = 2 represent the frame number.

FIG. 6 is a diagram explaining the processing of the data comparison processing unit 50. FIG. 6 shows an example of the first tracking information D 1 (left figure) generated by the first analysis section 30 and the second tracking information D 2 (right figure) generated by the second analysis section 40. There is. The first tracking information D 1 (left figure) and the second tracking information D 2 (right figure) in FIG. 6 are generated in the same time zone.

The image processing device 100 includes a first image acquisition unit 10, a second image acquisition unit 20, a first analysis unit 30, a second analysis unit 40, a data comparison processing unit 50, and a data output unit 60. I have it.

The first image acquisition unit 10 acquires image data of a range image generated by the laser radar 200. The first image acquisition unit 10 sequentially acquires distance images arranged in a time series from the laser radar 200.

The second image acquisition unit 20 is the image data of the camera image generated by the camera 300. 〇 2020/175085 7 卩(：171? 2020/004719

To get. The second image acquisition unit 20 sequentially acquires camera images arranged in time series from the camera 300.

The first analysis unit 30 tracks the position of the moving object existing in the monitoring target area based on the time-series distance images, and outputs the result as the first tracking information port 1.

[0035] Specifically, the first analysis unit 30 detects a moving object appearing in each frame of the distance images arranged in time series, gives a mouth to each moving object, and exists the moving object. The position is stored in association with the mouth. The first analysis unit 30 calculates, for example, the degree of association between the moving body detected in the frame of interest and the moving body detected in the previous frame, and based on the degree of association, the moving body detected in the frame of interest and the front Determine the identity with the moving object detected in the frame. Then, at this time, if the moving body detected in the target frame and the moving body detected in the previous frame are the same, the first analysis unit 30 detects that the moving body detected in the target frame is If the same edge as the moving body detected in the frame is given and the moving body detected in the target frame and the moving body detected in the previous frame are not the same, the moving body detected in the target frame is Assign a new I 0. In this way, the first analysis unit 30 individually tracks the moving objects reflected in each frame.

[0036] In the present embodiment, the first analysis unit 30 is provided for each physical body such as a person or a working machine.

I Give a mouth and identify its location. However, even if the first analysis unit 30 recognizes each part of one individual, such as a human arm, head, or leg, as a different moving body, and recognizes the position of each part of the individual. Good.

The method of detecting the moving body from the range image by the first analysis unit 30 may be any known method. The first analysis unit 30 may detect the moving object, for example, by taking the difference between the frame of interest and the previous frame. In addition, the first analysis unit 30 detects a moving object (for example, a person or a vehicle) by pattern matching based on, for example, the feature amount (for example, shape and size) of the cluster of distance measuring points in the distance image. You may.

[0038] Further, the first analysis unit 30 is a method for determining the identity of each moving object between different frames. May be any known method. The first analysis unit 30 may determine, for example, the distance between the moving object detected in the frame of interest and the moving object detected in the previous frame, the size similarity between the two, and the shape similarity between the two. The degree of relevance between the two is calculated based on the gender, the color similarity between the two, and the moving speed similarity between the two. Then, the first analysis unit 30 determines that the moving object detected in the frame of interest and the moving object detected in the previous frame are the same _ when the relationship is equal to or more than the predetermined value.

However, more preferably, the first analysis unit 30 uses a discriminator model that has been learned by machine learning to detect a moving object in the range image, or the first analysis unit 30 is different. Determine the identity of each moving object between frames. This makes it possible to enhance the robustness of moving body detection and the robustness of moving body identity determination against changes in the appearance of moving bodies. For example, when the first analysis unit 30 detects a moving body from the range image, it is preferable to use a convolutional neural network (Convolutional Neural Network: C NN). Further, when the first analysis unit 30 determines the identity of each moving object between different frames, it is preferable to use a hidden Markov model (Hidden Markov Mode l: HM M ), for example. At that time, for the classifier model, machine learning may be executed by reinforcement learning using a learning data set in which the input image and the correct answer value are associated with each other.

[0040] The first tracking information D1 is, for example, data indicating the position (here, three-dimensional coordinate position) of the moving body in each frame of the distance images arranged in time series (see Fig. 5A. See). The first tracking information D 1 includes, for example, for each moving body, an ID and information regarding a temporal change in the position of the moving body. In other words, the first tracking information D 1 identifies the moving body of the same ^{_ in} each frame by giving only one _ D. When there are multiple moving objects in the monitored area, the first tracking information D 1 is recorded by assigning an individual ID to each of the multiple moving objects and associating the temporal change of the position of the moving object. To do.

[0041] The second analysis unit 40 tracks the position of the moving object existing in the monitoring target area based on the time-sequential camera images, and outputs the result as the second tracking information. 〇 2020/175085 9 卩 (：171? 2020 /004719

Output as mouth 2.

[0042] As with the first analysis section 30, the second analysis section 40 detects a moving body from a camera image and the method of identifying each moving body among different frames is a known arbitrary method. Can be used. It is desirable that the second analysis unit 40 uses the trained discriminator model when performing these processes, similarly to the first analysis unit 30.

[0043] The second tracking information 02 is, for example, data indicating the position (here, two-dimensional coordinate position) of the moving object in each frame of the camera images arranged in time series (see Fig. 5). .. In the second tracking information 02, as with the first tracking information port 1, the mouth and the temporal change of the position of the moving body are recorded in association with each other for each moving body.

The data comparison processing unit 50 compares the first tracking information port 1 and the second tracking information port 2 while aligning the time axes, and determines the first tracking information port 1 and the second tracking information port 0 2. Based on this, the position and I 0 of the moving object existing in the monitored area are confirmed, and the result is output as the confirmed tracking information port 3.

[0045] In the left diagram of Fig. 6, an error occurred in the moving object identification processing in the first analysis unit 30 and the mouth to be given to the same moving object (here, a person) was changed on the way. The mode (1 0 ^ 1 1 ^ 1 0 ^ 12) is shown. On the other hand, the right diagram of Fig. 6 shows a mode in which the second analysis unit 40 accurately captures the I mouth of the same moving body (here, a person).

[0046] In general, it is possible to specify the position of the moving body not only in the horizontal and vertical directions but also in the depth direction from the range image. Therefore, specifying the position of each part of the moving body based on the distance image is useful in accurately specifying the posture of the moving body and the behavior of the moving body.

However, unlike a camera image, a range image has low resolution and does not include color information. Therefore, it may be difficult to identify a moving object between different frames from the range image. In particular, when tracking a moving object, if the moving object intersects another moving object or changes its posture, whether the moving object is the same as the one detected at the previous time point. Can't identify 〇 2020/175085 10 boxes (：171? 2020 /004719

There is

[0048] On the other hand, since the camera image has high resolution and includes color information, it is possible to more accurately identify a moving object between different frames.

Therefore, the data comparison processing unit 50 uses the second tracking information 02 to determine the first

1 Determine the correctness of the I mouth of the moving object included in the tracking information mouth 1. Then, the data comparison processing unit 50, if the change in the index 0 of the moving object included in the first tracking information port 1 is different from the change in the I 0 of the moving object included in the second tracking information port 2, Considering that there is an error in the change of the mouth I of the moving object included in tracking information port 1, I 0 of the moving object included in first tracking information port 1 is changed to I 0 of the moving object included in second tracking information port 2. After correction, the corrected first tracking information 0 1 is output as the fixed tracking information port 3.

[0050] As a method for the data comparison processing unit 50 to identify the correspondence between the moving body included in the first tracking information port 1 and the moving body included in the second tracking information port 2, typically, the first tracking A method of calculating the degree of proximity between the position of the moving body included in the information port 1 and the position of the moving body included in the second tracking information 02 (for example, the distance between the two or the overlapping area of the two) is used. In order that the data comparison processing unit 50 can execute the process, a data set indicating the correspondence between the coordinate position indicated by the first tracking information port 1 and the coordinate position indicated by the second tracking information port 2 is displayed. It is desirable that the image is stored in advance in the image processing device 100 such as [1/1\/110 2].

[0051] However, the data comparison processing unit 50 refers to the visual field information of the laser radar 200 and the visual field information of the camera 300 to consider the difference between the visual fields of the two, and then performs the first tracking. It is desirable to identify the correspondence between the moving object included in information port 1 and the moving object included in the second tracking information port 2.

[0052] Fig. 7 is a diagram showing the field of view of the laser radar 200 and the field of view of the camera 300. The field of view 1 of the laser radar 200 is generally narrower in the vertical imaging range than the field of view 2 of the camera 300. Therefore, the moving object shown in the range image is different from the moving object shown in the camera image, and the whole moving object is shown in the range image. 〇 2020/175085 1 1 卩(：171? 2020/004719

It may not be. For example, a camera image may show the whole body of a person, while a range image may show only the upper half of the person. Therefore, the data comparison processing unit 50, for example, considers the difference in shape between the moving object in the range image and the moving object in the camera image, and considers the moving object included in the first tracking information port 1 and the second object. It is desirable to identify the correspondence with the moving object included in tracking information port 2. Comparison unit 5 0, for example, becomes large is the same ^_ object conditions vertically to more if the object being Mikire completely vertically angle in the laser radar 2 0 0, the laser radar 2 0 0 If an object is completely contained within the vertical angle of view of, it can be judged that an object larger than that is likely to be another object even if it is approximate in terms of distance. This can improve the accuracy of object discrimination.

On the other hand, from the viewpoint of reducing the processing load, the data comparison processing unit 50 more preferably refers to the first tracking information port 1 and refers to any one of the following first to third situations. It is desirable to judge whether or not it has occurred and refer to the second tracking information port 2 only when any of the first to third situations has occurred. In other words, the data comparison processing unit 50 determines the first tracking information port 1 as it is without referring to the second tracking information port 2 when neither of the first situation to the third situation occurs. It is desirable to output as tracking information 0 3.

[0054] Specifically, in the first situation, in the first tracking information port 1, in the same timing, one door disappears and a new, separate one is generated. The second situation is that one moving body intersects another moving body at the first tracking information port 1. The third situation is that in the first tracking information port 1, a certain moving object once disappears from within the range image, and then appears again as a moving object of the same opening.

[0055] If any of the first situation to the third situation occurs in the first tracking information port 1, an error has occurred in 0 of the moving object included in the first tracking information port 1. there is a possibility. For example, in the first situation, the same moving body 〇 2020/175085 12 boxes (：171? 2020 /004719

— There is a possibility that the frame and the other frame are erroneously recognized as different moving objects. Also, in the second situation, at the timing when a certain moving body intersects with another moving body, it is possible that the certain moving body is erroneously recognized as another moving body that intersected. Also, in the third situation, from the position where one moving body disappears from the monitored area, it is possible that another moving body that has entered the monitored area is erroneously recognized as the previously existing moving body. Yes (or vice versa). On the other hand, if none of the first to third situations has occurred, it is unlikely that an error has occurred in the mouth of the moving object included in the first tracking information port 1.

From this point of view, it is desirable that the data comparison processing unit 50 decides whether or not to carry out the correctness determination of the first tracking information port 1 depending on the situation as described above. As a result, unnecessary data processing can be omitted and the processing load can be reduced.

The data output unit 60 outputs the definite tracking information 03 generated by the data comparison processing unit 50 to the outside. The data output unit 60 converts, for example, the confirmed tracking information port 3 into display image format data and displays it on the display.

[Operation Flow of Image Processing Device]

Next, with reference to FIG. 8, an example of the operation of the image processing apparatus 100 according to the present embodiment will be described.

FIG. 8 is a flow chart showing processing executed when the image processing apparatus 100 (data comparison processing unit 50) according to the present embodiment generates fixed tracking information. The flow chart shown in FIG. 8 is executed by the image processing apparatus 100 according to a computer program, for example.

[0060] Steps 31 to 35 are preprocessing steps for performing a process of comparing the first tracking information port 1 and the second tracking information port 2. In this pre-processing step, first, the image processing device 100 (first analysis unit 30) generates the first tracking information port 1 based on the time-sequential distance images (step 31). .. Next, the image processing device 100 (second analysis unit 40) was used to display the camera images arranged in time series. 〇 2020/175085 13 卩 (：171? 2020 /004719

Second tracking information 0 2 is generated based on the image (step 3 2 ). Next, the image processing device 100 sets the analysis target time (for example, time D = 8: 0 0: 0 0) for comparing the first tracking information port 1 and the second tracking information port 2 (step 3 3). Next, the image processing device 100 sets the number of frames to be analyzed when comparing the first tracking information port 1 and the second tracking information 0 2 (for example, the number of frames n = 5) ( Step 3 4). Next, the image processing device 100 sets the time interval of the analysis target (for example, analysis target interval = 1 second interval) (step 35).

[0061] Next, the image processing apparatus 100 causes the storage unit for data comparison (for example, the comparison queue) to store the first frame of one frame according to the conditions set in steps 33 to 35. Register tracking information port 1 and second tracking information 0 2 (step 36). Next, the image processing apparatus 100 determines whether or not the storage section includes two or more frames of data (first tracking information port 1 and second tracking information 0 2) (step 3 7), if data of 2 frames or more is included {~1 :丫巳 3), proceed to the determination process of the following step 38, while

When the data of 2 frames or more is not included (37: N 0), the process proceeds to the following step 3 1 2.

Next, the image processing apparatus 100 uses the first tracking information port stored in the storage unit.

It is determined whether 1 includes the disappearance and generation of the mouth in the same timing (step 38), and if the disappearance and generation of the mouth I is included (3 8: 巫3 3), continue. Then proceed to step 39. On the other hand, if the extinction and the creation of the mouth are not included (3 8 :N 0), the process proceeds to the following step 3 1 2.

[0063] Next, the image processing apparatus 100 determines whether or not the moving body of the disappearance I0 extracted in step 38 and the moving body of the newly generated entrance satisfy predetermined conditions regarding position and size. However, if the predetermined condition is satisfied (39: No. 3), the process proceeds to the following step 310, and if the predetermined condition is not satisfied (39: N 0), the next step. Proceed to 3 1 2. Specifically, the image processing apparatus 100 uses the position of the center of gravity of the moving body of the vanishing I mouth extracted in step 38 and the new position. 〇 2020/175085 14 卩 (：171? 2020 /004719

Whether the distance between the center of gravity of the moving object and the center of gravity of the moving object is less than or equal to a threshold value (for example, within 100 pixels) in the image, and the ratio between the sizes is within a predetermined range (for example, 0.8 1.2) is determined.

Next, the image processing apparatus 100 uses the second tracking information port stored in the storage unit.

Second, it is determined whether or not the mouth of the moving body is switched near the position where the disappearance and generation of the I mouth of the moving body are detected in the first tracking information port 1 (step 310). Then, the image processing apparatus 100 considers that there is no error in the 0 of the 1st tracking information port 1 when the switching of the moving image's head has occurred (3 10: Togomi 3), The first tracking information port 1 stored in the storage unit is directly output as the finalized tracking information port 3 (steps 3 1 2). On the other hand, it is considered that the image processing device 100 has an error in the 0 of the 1st tracking information port 1 when the switching of the moving image's port does not occur (3 10: 1\100). Then, after correcting I 0 of the moving object of the 1st tracking information port 1 stored in the storage unit to the original I 0 of the 1st tracking information port 1 (Step 3 1 1), the corrected 1st tracking information is , Definite tracking information port 3 is output (step 3 1 2).

With this processing, the correction processing of the first tracking information port 1 for the number of frames set in step 34 is completed. Then, the image processing apparatus 100 advances the analysis target time (Cing +) (step 3 13) in order to execute the correction process for the first tracking information port 1 at the subsequent time. Next, the image processing apparatus 100 determines whether or not there is unprocessed data at the first tracking information port 1 output by the first analysis unit 30 (step 3 14), and the unprocessed data is processed. If there is data (3 1 4 :丫巳 3), return to step 36 and perform the same processing. On the other hand, when there is no unprocessed data (3 14: N 0), the series of flow chart processing is terminated.

[0066] [Effect]

As described above, in the image processing apparatus 100 according to the present embodiment, the data comparison processing unit 50 stores the first tracking information port 1 and the second tracking information 02. 〇 2020/175085 15 卩(：171? 2020/004719

The axes are aligned and compared, and the position of the moving object and the I port existing in the monitored area are determined based on the first tracking information port 1 and the second tracking information port 2, and the result is confirmed. Output as 3.

Therefore, according to the image processing apparatus 100 according to the present embodiment, it is possible to improve the identification performance of the moving object existing in the monitoring target area. As a result, the accuracy of moving object tracking using the laser radar 200 can be improved.

[0068] (Second Embodiment)

Next, an image processing device 100 according to the second embodiment will be described with reference to FIG. FIG. 9 is a diagram showing the configuration of the image processing apparatus 100 according to the second embodiment.

The image processing device 100 according to this embodiment is different from that of the first embodiment in that it is built in the laser radar 200. Then, the first image acquisition unit 10 of the image processing device 100 acquires image data directly from the range image generation unit 210 (that is, the imaging unit that generates the range image) of the laser radar 200.

According to the image processing apparatus 100 according to the present embodiment, it is possible to eliminate the need to prepare a separate computer other than the laser radar 200 and the camera 300.

[0071] (Third Embodiment)

Next, an image processing apparatus 100 according to the third embodiment will be described with reference to FIG. FIG. 10 is a diagram showing the configuration of the image processing apparatus 100 according to the third embodiment.

The image processing apparatus 100 according to the present embodiment acquires a power camera image from a thermal camera 400 instead of the camera 300 (hereinafter referred to as “visible light camera”) of the above embodiment. The configuration is different from that of the first embodiment.

Since the camera image generated by the visible light camera 300 includes color information, high object identification accuracy can be realized. On the other hand, when the monitored area is covered with fog or at night, the object identification accuracy using camera images is low. 〇 2020/175085 16 卩(：171? 2020/004719

Down.

In this respect, since the thermal camera 400 generates a camera image based on infrared light emitted from an object, it is possible to realize high object identification accuracy even in a situation such as at night. Therefore, the image processing device 100 (second analysis unit 40) according to the present embodiment generates the second tracking information 02 by using the camera image generated by the thermal camera 400. Then, the image processing apparatus 100 (data comparison processing unit 50) according to the present embodiment uses the second tracking information 02 to correct an error in the first tracking information window 1.

As described above, according to the image processing apparatus 100 according to the present embodiment, moving object tracking is executed with high accuracy even in a situation where the monitored area is covered with fog or at night. be able to.

The image processing apparatus 100 according to this embodiment may use both the camera image generated by the visible light camera and the camera image generated by the thermal camera 400. Further, when the second tracking information 02 is generated, both the camera image generated by the visible light camera and the camera image generated by the thermal camera 400 may be used, or selectively. The camera images used may be switched.

[0077] (Other Embodiments)

The present invention is not limited to the above-described embodiment, and various modifications can be considered.

[0078] For example, in the above-described embodiment, a mode in which a D-type laser radar is used is shown as an example of the laser radar 200, but the configuration of the laser radar 200 is arbitrary. For example, as the laser radar 200, a 1\/1 〇 \/\/ type laser radar or the like may be used.

Further, in the above embodiment, as an example of the configuration of the image processing device 100, the first image acquisition unit 100, the second image acquisition unit 20, the first analysis unit 30 and the second analysis unit are used. Although the functions of the data processing unit 40, the data comparison processing unit 50, and the data output unit 60 are described as being realized by one computer, the image processing device 100 is realized by a plurality of computers. Of course, it is okay. Also, the computer 〇 2020/175085 17 卩(：171? 2020/004719

The programs and data that are read out may also be distributed and stored in multiple computers.

Although specific examples of the present invention have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and changes of the specific examples illustrated above.

[0081] The disclosures of the specification, drawings and abstract included in the Japanese application of Japanese Patent Application No. 201-9-032921 filed on Feb. 26, 2010 are incorporated herein by reference in their entirety. Industrial availability

According to the image processing device of the present disclosure, it is possible to improve the accuracy of moving body tracking using a laser radar.

Explanation of symbols

[0083] II monitoring system

1 00 Image processor

1 01 〇 II

1 04 External storage device

1 05 Communication interface

1 0 1st image acquisition unit

20 Second image acquisition unit

30 1st analysis section

40 2nd Analysis Department

50 Data comparison processing unit

60 Data output section

200 laser radar

2 1 0 Distance image generator

300 cameras (visible light camera)

400 thermal camera \¥0 2020/175085 18 卩 (: 17 2020 /004719

〇 1 1st tracking information

0 2 Second tracking information

0 3 Confirmed tracking information

Claims

〇 2020/175085 19 卩(：171? 2020/004719 Claims

[Claim 1] A first image acquisition unit that acquires image data of a range image from a laser radar that monitors a predetermined area,

A first analysis unit that generates first tracking information by tracking the position of a moving object existing in the predetermined area based on the distance images arranged in time series,

A second analysis unit that generates second tracking information by tracking the position of the moving body existing in the predetermined area based on the camera images arranged in time series,

The first tracking information and the second tracking information are compared on the same time axis, and the identification information and the position of the moving object existing in the predetermined area are determined based on the first tracking information and the second tracking information. A data comparison processing unit that

An image processing apparatus including.

2. The image processing apparatus according to claim 1, wherein the data comparison processing unit corrects the moving body identification information included in the first tracking information based on the second tracking information.

[Claim 3] When the moving object existing in the predetermined area corresponds to a predetermined condition, the data comparison processing unit determines whether the moving object exists in the predetermined area based on the first tracking information and the second tracking information. Confirm the identification information and position of the existing moving body,

If the predetermined situation is not met, the identification information and the position of the moving object existing in the predetermined area are determined based on the first tracking information without referring to the second tracking information,

The image processing apparatus according to claim 1.

[Claim 4] The predetermined situation is the same tracking in the first tracking information. 〇 2020/175085 20 boxes (: 171-1? 2020 /004719

In imming, a situation in which one piece of identification information is lost and a new piece of identification information is newly generated,

The image processing apparatus according to claim 3.

[Claim 5] The predetermined situation includes a situation in which one moving body intersects with another moving body in the first tracking information.

The image processing apparatus according to claim 3 or 4.

[Claim 6] The predetermined situation includes a situation in which, in the first tracking information, one moving body once appears as a moving body of the same identification information after once disappearing from within the range image. ,

The image processing apparatus according to claim 3.

7. The data comparison processing unit, based on the information on the degree of proximity or size between the position of the moving body included in the first tracking information and the position of the moving body included in the second tracking information, By identifying the correspondence between the moving body included in the first tracking information and the moving body included in the second tracking information, the identification information and the position of the moving body existing in the predetermined area are determined.

The image processing device according to claim 1.

[Claim 8] The data comparison processing unit determines, based on the visual field information of the laser radar and the visual field information of the camera, a moving body included in the first tracking information and a moving body included in the second tracking information. Specify the correspondence of

The image processing apparatus according to claim 7.

9. The first analysis unit and/or the second analysis unit uses a discriminator model that has been learned by machine learning to track the position of a moving object existing in the predetermined region,

The image processing device according to claim 1.

[Claim 10] The camera is a visible camera,

The image processing device according to claim 1. 〇 2020/175085 21 卩 (：171? 2020 /004719

[Claim 11] The camera is a thermal camera,

The image processing device according to any one of claims 1 to 10.

[Claim 12] The camera includes a visible camera and a thermal camera,

The image processing apparatus according to any one of claims 1 to 11.

[Claim 13] Obtaining image data of a range image from a laser radar that monitors a predetermined area,

Obtaining image data of a camera image from a camera monitoring the predetermined area,

Based on the distance images arranged in chronological order, the first tracking information that tracks the position of the moving object existing in the predetermined area is generated, and based on the camera images arranged in chronological order, the moving object existing in the predetermined area is generated. Second tracking information is generated by tracking the position of the first tracking information, and the first tracking information and the second tracking information are compared with each other on the same time axis, and based on the first tracking information and the second tracking information, Confirm the identification information and position of the moving object existing in the predetermined area,

Image processing method.