JP2021111195A - Image processing device and image processing method - Google Patents

Abstract

To provide an image processing device and an image processing method for matching objects in a picked-up image with high accuracy.SOLUTION: An image processing device 1 includes: an image acquisition unit 22 for acquiring a picked-up image with a plurality of objects as subjects at a predetermined interval; a first position specification unit 23 for specifying a first position of an image coordinate system of the plurality of objects included in the picked-up image; a second position specification unit 24 for specifying a second position of a three-dimensional coordinate system of the plurality of objects; a dissimilarity calculation unit 25 for acquiring a first dissimilarity in the image coordinate system and a second dissimilarity in the three-dimensional coordinate system, respectively on the basis of the first position and the second position of a plurality of tracking objects specified as tracking objects at one timing, and the first position and the second position of the plurality of objects of the picked-up image acquired by the image acquisition unit 22 at timing after the one timing; and a matching processing unit 26 for performing matching processing of each object of the picked-up image to the tracking objects on the basis of both the acquired first dissimilarity and the second dissimilarity.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus and an image processing method.

Tracking processing is performed to recognize that the objects of a plurality of captured images captured at different times are the same object. For example, it is determined whether or not the objects are the same by performing the matching process by obtaining the degree of difference (or the degree of similarity) between the objects in the image coordinate system.

International Publication No. 2019/16879

By the way, a plurality of objects may be included in the captured image to be matched. Depending on the positional relationship of a plurality of objects in the captured image, each object cannot be properly distinguished, and a mismatch may occur. For example, when a plurality of objects are positioned so as to overlap each other in the captured image, it is difficult to distinguish the objects only by the position information on the captured image. On the other hand, when the distance (depth) information of an object is calculated in the image processing and the matching processing is performed based on the information, a mismatch may occur due to the noise included in the distance.

Therefore, the present invention has been made in view of these points, and an object of the present invention is to match objects in a captured image with high accuracy.

In the first aspect of the present invention, there is an image coordinate system of an image acquisition unit that acquires captured images of a plurality of objects as subjects at predetermined intervals and a plurality of objects corresponding to each object included in the captured images. The first position specifying unit that specifies the first position, which is the position of, and the positions of the plurality of objects in the depth direction in the captured image are estimated, and the positions of the plurality of objects in the three-dimensional coordinate system. The second position specifying unit that specifies the second position, the first position and the second position of a plurality of tracking objects specified as tracking targets at one timing, and the image acquisition unit at a timing later than one. Based on the first position and the second position of a plurality of objects in the captured image acquired by, the difference for obtaining the first difference degree in the image coordinate system and the second difference degree in the three-dimensional coordinate system, respectively. An image including a degree calculation unit and a matching processing unit that performs matching processing of each object of the captured image with respect to the tracking object based on both the obtained first difference degree and the second difference degree. A processing device is provided.

Further, the difference degree calculation unit may make the weighting value for the first difference degree different from the weighting value for the second difference degree.

Further, the difference degree calculation unit may also obtain the difference degree using information based on the size or the brightness value of the tracking object and each object of the captured image.

Further, the image acquisition unit may acquire a front image obtained by capturing a plurality of objects traveling in front of the vehicle as the captured image.

In the second aspect of the present invention, there is a step of acquiring captured images of a plurality of objects as subjects at predetermined intervals, and positions of a plurality of objects corresponding to each object included in the captured images in an image coordinate system. The step of specifying the first position, and the position of the plurality of objects in the depth direction in the captured image are estimated, and the second position, which is the position of the plurality of objects in the three-dimensional coordinate system, is specified. The first position and the second position of the plurality of tracking objects specified as tracking targets at one timing, and the first position and the first position of the plurality of objects of the captured image acquired at the timing after one. A step of obtaining the first difference degree in the image coordinate system and the second difference degree in the three-dimensional coordinate system, respectively, based on the second position, and the obtained first difference degree and the second difference degree. Provided is an image processing method including a step of performing matching processing of each object of the captured image with respect to the tracking object based on both of the above.

According to the present invention, there is an effect that objects in a captured image can be matched with high accuracy.

It is a schematic diagram for demonstrating the structure of the image processing apparatus 1 which concerns on one Embodiment. It is a schematic diagram for demonstrating the outline of the matching process of the image processing apparatus 1. It is a schematic diagram for demonstrating the position of an object in an image coordinate system. It is a schematic diagram for demonstrating the position of an object in a three-dimensional coordinate system. It is a schematic diagram which shows the score table. It is a flowchart for demonstrating the flow of a matching process.

<Configuration of image processing device>
The configuration of the image processing apparatus according to the embodiment of the present invention will be described with reference to FIG.

FIG. 1 is a schematic diagram for explaining the configuration of the image processing device 1 according to the embodiment. The image processing device 1 performs matching processing of objects included in the captured images sequentially captured by the imaging device 5. The image processing device 1 is mounted on a vehicle, for example, and the matching process described above is used, for example, to track a vehicle in front of or around the vehicle during automatic driving.

The image pickup device 5 is mounted on a vehicle, for example, and images the front of the vehicle. Specifically, the image pickup device 5 images another vehicle traveling in front of the vehicle while the vehicle is traveling. The image pickup device 5 transmits the captured captured image to the image processing device 1. The image pickup device 5 is provided, for example, in the upper part of the front of the vehicle, and images an obliquely front portion. Therefore, when the vehicles are traveling in a row in front of the vehicle, the images are taken so that the vehicles overlap. The image pickup device 5 may be mounted on the side surface or the rear surface of the vehicle, and may image a direction other than the front.

FIG. 2 is a schematic diagram for explaining the outline of the matching process of the image processing apparatus 1. Here, the objects M1 to M4 of the captured image P2 captured at time k + 1 are the observation objects to be matched. The image processing device 1 performs matching processing of the objects M1 to M4 of the captured image P2 on the tracking object that has already been identified. The tracking objects are, for example, objects L1 to L4 whose IDs have already been specified by performing matching processing on the captured image P1 captured at time k. The tracking object may include an object specified before the time k, and is stored in the storage unit 10 described later.
Although the details will be described later, the image processing device 1 searches for the same object as the tracking object in the observed object of the captured image P2 by obtaining the degree of difference in both the image coordinate system and the three-dimensional coordinate system. This makes it possible to match objects with high accuracy.

As shown in FIG. 1, the image processing device 1 has a storage unit 10 and a control unit 20.
The storage unit 10 includes, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 10 stores a program and various data for execution by the control unit 20. For example, the storage unit 10 stores information on an algorithm for performing matching processing, which will be described later, and information on a tracking object whose ID has already been specified.

The control unit 20 is, for example, a CPU (Central Processing Unit). The control unit 20 executes a program stored in the storage unit 10 to perform matching processing of objects included in the captured image (specifically, the captured image P2 in FIG. 2). In the present embodiment, the control unit 20 functions as an image acquisition unit 22, a first position identification unit 23, a second position identification unit 24, a difference degree calculation unit 25, and a matching processing unit 26.

The image acquisition unit 22 acquires captured images of a plurality of objects as subjects at predetermined intervals. That is, the image acquisition unit 22 sequentially acquires the captured images captured by the imaging device 5. For example, the image acquisition unit 22 acquires the captured image P2 shown in FIG. Here, the image acquisition unit 22 acquires a front image of a plurality of objects traveling in front of the vehicle as an image to be captured.

The first position specifying unit 23 specifies the first position, which is the position of a plurality of objects corresponding to each object included in the captured image (for example, the captured image P2 in FIG. 2) in the image coordinate system of the captured image. do. That is, the first position specifying unit 23 specifies the first position of each object in the captured image acquired by the image acquisition unit 22.

FIG. 3 is a schematic diagram for explaining the position of the object in the image coordinate system. Here, it is assumed that four objects A, B, C, and D are included in the captured image. For example, the four objects A to D are other vehicles traveling in front of each other. Further, in the image coordinate system, here, it is assumed that the upper left end of the captured image is the origin, the horizontal direction of the captured image is the x-axis direction, and the vertical direction of the captured image is the y-axis. The first position specifying unit 23 specifies the positions (positions in the x and y axis directions) of the objects A to D in the above image coordinate system as the first position.

The second position specifying unit 24 estimates the positions of a plurality of objects in the depth direction in the captured image by geometrical constraint conditions or integration of information with other sensors, and in the three-dimensional coordinate system of the plurality of objects. Each second position, which is a position, is specified. Further, in the three-dimensional coordinate system, the camera position is taken as the origin, the right hand direction of the vehicle is the X-axis direction, the height direction is the Y-axis, and the front direction of the vehicle is the Z-axis.

FIG. 4 is a schematic diagram for explaining the position of an object in a three-dimensional coordinate system. In FIG. 4, after specifying the depth coordinates (Z-axis) of each object A to D in the captured image of FIG. 3, the X-coordinate and the Y-coordinate are calculated, and their positions are drawn as a bird's-eye view. The second position specifying unit 24 specifies the positions (positions in the X, Y, and Z axis directions) of the objects A to D in the above three-dimensional coordinate system as the second position.

The difference degree calculation unit 25 is the first position and the second position of the tracking object specified as the tracking target at one timing, and the second of a plurality of objects of the captured image acquired by the image processing unit 22 at the timing after one. Based on the first position and the second position, the first degree of difference in the image coordinate system and the second degree of difference in the three-dimensional coordinate system are obtained, respectively. That is, the difference degree calculation unit 25 obtains the first difference degree and the second difference degree based on the first position and the second position specified by the first position specifying unit 23 and the second position specifying unit 24. For example, the difference calculation unit 25 is based on the first position and the second position of the tracking object (objects L1 to L4 in FIG. 2) and the first position and the second position of the objects M1 to M4 of the captured image P2. The first degree of difference and the second degree of difference are obtained.

上記の式において、ｘ_trackが追跡オブジェクトの画像座標系でのｘ座標を意味し、ｘ_obsが観測オブジェクトの画像座標系でのｘ座標を意味する。同様に、Ｘ_trackが追跡オブジェクトの三次元座標系でのＸ座標を意味し、Ｘ_obsが観測オブジェクトの三次元座標系でのＸ座標を意味する。λ_imageとλ_3dは、それぞれ画像座標系と三次元座標系でのスコアの重み係数であり_、λ_ｘ、λ_Ｘ等が各座標値に対する重み係数である。なお、各座標値に対する重み係数のみで画像座標系と三次元座標系に対する影響度は調整可能なため、λ_imageとλ_3dは無くてもよい。また、上記の式では、簡単にするため、比較対象のオブジェクト間の距離のＬ１ノルムの各成分の重み付き和をスコアとしているが、ユークリッド距離、すなわち、Ｌ２ノルムや確率的な位置の分布を考慮したマハラノビス距離などをスコアとして採用してもよい。相違度算出部２５は、全ての追跡オブジェクトと、撮像画像Ｐ２の全ての観測オブジェクトについて、スコア関数を求める。 In the present embodiment, the difference calculation unit 25 obtains a score function indicating the first difference and the second difference. The score function Score is a score obtained by scoring the difference in coordinates between an object acquired at one timing or an object being tracked and an observed object. For example, the following formula is used as the score between one tracking object and one observation object.

In the above equation, x _track means the x coordinate of the tracking object in the image coordinate system, and x _obs means the x coordinate of the observed object in the image coordinate system. Similarly, X _track means the X coordinate of the tracking object in the 3D coordinate system, and X _obs means the X coordinate of the observed object in the 3D coordinate system. lambda _image and lambda _3d is a weight factor of the scores of each image coordinate system and the three-dimensional coordinate _system, lambda _x, lambda _X, etc. is a weighting factor for each coordinate value. Since the degree of influence on the image coordinate system and the three-dimensional coordinate system can be adjusted only by the weighting coefficient for each coordinate value, λ _image and λ _3d may be omitted. Further, in the above equation, for the sake of simplicity, the weighted sum of each component of the L1 norm of the distance between the objects to be compared is used as the score, but the Euclidean distance, that is, the distribution of the L2 norm and the stochastic position is used. The Mahalanobis distance taken into consideration may be adopted as the score. The difference calculation unit 25 obtains a score function for all the tracking objects and all the observed objects of the captured image P2.

ここで、ｗとｈはそれぞれ画像座標系におけるオブジェクトの幅と高さであり、ＷとＨはそれぞれ三次元座標系におけるオブジェクトの幅と高さである。 The difference degree calculation unit 25 may obtain the difference degree based on the position of each object of the first captured image and the second captured image, as well as the size, texture information, and the like. For example, when considering the position and size of each object, the score function is calculated as follows.

Here, w and h are the width and height of the object in the image coordinate system, respectively, and W and H are the width and height of the object in the three-dimensional coordinate system, respectively.

Further, the difference degree calculation unit 25 may make the value of the weighting for the first degree of difference (for example, λ _image , λ _x ) different from the value of the weighting for the second degree of difference (for example, λ _3d , λ _X ). .. For example, the difference calculation unit 25 sets the weighting value for the second difference to be larger than the weighting value for the first difference in order to emphasize the calculation of the difference in the three-dimensional coordinate system.

The matching processing unit 26 performs matching processing of each object (observation object) of the captured image on the tracking object based on both the obtained first difference degree and the second difference degree. For example, when both the first difference degree and the second difference degree are small, the matching processing unit 26 determines that the tracking object for which the difference degree is obtained and the observation object are the same object. On the other hand, when the degree of difference is large in either the first degree of difference or the second degree of difference, the matching processing unit 26 determines that the tracking object for which the degree of difference is obtained and the observation object are different objects.

FIG. 5 is a schematic diagram showing a score table. Here, the score table is a table that summarizes the score values of the score functions obtained for each of the tracking objects L1 to L4 and the observation objects M1 to M4 shown in FIG. The matching processing unit 26 searches for an observation object that can be regarded as the same as each tracking object by using the score table. For example, the matching processing unit 26 searches for an observation object having a small score value from the score values S11 to S14 with each of the four observation objects M1 to M4 with respect to the tracking object L1. When there are a plurality of tracking objects, the matching processing unit 26 assigns the optimum observation object to each tracking object by using, for example, the Hungarian method. The Hungarian method is an algorithm that uniquely determines the optimal set. As a result, the optimum combination of a plurality of tracking objects and observation objects can be realized.

<Flow of matching process>
The flow of the matching process will be described with reference to FIG.

FIG. 6 is a flowchart for explaining the flow of the matching process. Here, the matching process of the tracking object and each object of the captured image P2 will be described. The tracking object is identified by, for example, a matching process after acquisition of the captured image P1 and stored in the storage unit 10.

First, the image acquisition unit 22 acquires an captured image including the observation object (for example, the captured image P2 shown in FIG. 2) (step S102).

Next, the first position specifying unit 23 specifies the first position of the observed object of the captured image P2 in the image coordinate system (step S104). Further, the second position specifying unit 24 specifies the second position of the observed object of the captured image P2 in the three-dimensional coordinate system (step S106).

Next, the difference degree calculation unit 25 obtains the first difference degree and the second difference degree of the tracking object and the observation object of the captured image P2 (step S108). For example, the difference calculation unit 25 obtains the score value of the score function including the first difference and the second difference between one tracking object and one observation object. Then, the difference calculation unit 25 derives, for example, the score table shown in FIG. 5 by obtaining the score values between all the tracking objects and the observed objects.

Next, the matching processing unit 26 matches the observed objects of the captured image P2 based on the obtained first difference degree and second difference degree (step S110). For example, the matching processing unit 26 searches for the same observation object as each tracking object using the score table shown in FIG. That is, the matching processing unit 26 determines that the tracking object and the observation object having a small calculated score are the same object. The matching processing unit 26 determines that the observation object whose difference degree is equal to or higher than the threshold value with any of the tracking objects is a new object. On the other hand, among the tracking objects, the object whose difference degree is equal to or more than the threshold value from any of the observed objects is discarded as a nonexistent object if the requirement for holding the object is not satisfied.

The image processing device 1 performs the matching process described above each time the image pickup device 5 captures the captured image. For example, when the next captured image of the captured image P2 is acquired, the object stored as the tracking object as a result of the matching process with the object of the captured image P2 and the object (observation object) in the next captured image Matching processing is performed between.

In the above description, the image processing device 1 performs tracking processing on the captured image captured by the image pickup device 5 mounted on the vehicle, but the present invention is not limited to this. For example, the image processing device 1 may perform tracking processing on an captured image captured by a camera installed in equipment other than a vehicle such as a surveillance camera.

Further, in the above, the score function is defined by the degree of difference between objects (tracking object and observation object), but the present invention is not limited to this, and for example, the score function may be defined by the degree of similarity between objects. When the score is defined by similarity, the tracking object with the higher score and the observed object are considered to be the same object.

<Effect in this embodiment>
The image processing device 1 of the above-described embodiment has a first degree of difference in the image coordinate system between the plurality of tracking objects already specified as tracking targets and the observation object of the captured image P2 acquired by the image acquisition unit 22. Find the second degree of difference in the three-dimensional coordinate system. The image processing device 1 performs matching processing of each observation object and each tracking object of the captured image P2 based on the obtained first difference degree and second difference degree.
By performing matching processing using the degree of difference between the two coordinate systems of the image coordinate system and the three-dimensional coordinate system, mismatching is prevented and accuracy is improved compared to the case where matching processing is performed using only the degree of difference of one coordinate system. Observed objects can be matched properly.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes can be made within the scope of the gist thereof. be. For example, all or a part of the device can be functionally or physically distributed / integrated in any unit. Also included in the embodiments of the present invention are new embodiments resulting from any combination of the plurality of embodiments. The effect of the new embodiment produced by the combination also has the effect of the original embodiment.

1 Image processing device 22 Image acquisition unit 23 1st position identification unit 24 2nd position identification unit 25 Difference degree calculation unit 26 Matching processing unit

Claims (5)

An image acquisition unit that acquires captured images of multiple objects at predetermined intervals,
A first position specifying unit that specifies a first position, which is a position in the image coordinate system, of a plurality of objects corresponding to each object included in the captured image.
A second position specifying unit that estimates the positions of the plurality of objects in the depth direction in the captured image and specifies the second positions that are the positions of the plurality of objects in the three-dimensional coordinate system, respectively.
The first position and the second position of the plurality of tracking objects specified as tracking targets at one timing, and the first position of the plurality of objects of the captured image acquired by the image acquisition unit at a timing after one. And a difference calculation unit that obtains the first difference in the image coordinate system and the second difference in the three-dimensional coordinate system based on the second position.
A matching processing unit that performs matching processing for each object of the captured image with respect to the tracking object based on both the obtained first difference degree and the second difference degree.
An image processing device. The difference calculation unit makes the weighting value for the first difference different from the weighting value for the second difference.
The image processing apparatus according to claim 1. The difference degree calculation unit also obtains the difference degree using information based on the size or brightness value of the tracking object and each object of the captured image.
The image processing apparatus according to claim 1 or 2. The image acquisition unit acquires a front image of a plurality of objects traveling in front of the vehicle as the captured image.
The image processing apparatus according to any one of claims 1 to 3. Steps to acquire captured images of multiple objects at predetermined intervals,
A step of specifying a first position, which is a position in the image coordinate system, of a plurality of objects corresponding to each object included in the captured image.
A step of estimating the position of the plurality of objects in the depth direction in the captured image and specifying a second position which is a position of the plurality of objects in the three-dimensional coordinate system, respectively.
The first position and the second position of the plurality of tracking objects specified as tracking targets at one timing, and the first position and the second position of the plurality of objects of the captured image acquired at the timing after one. Based on the above, the step of obtaining the first degree of difference in the image coordinate system and the second degree of difference in the three-dimensional coordinate system, respectively, and
A step of performing matching processing of each object of the captured image with respect to the tracking object based on both the obtained first difference degree and the second difference degree.
An image processing method having.

Images