JP6871199B2 - Target recognition device - Google Patents

Description

The present invention relates to a target object recognition device, and more particularly to a device that recognizes a target object shown in an image.

Research and development is being carried out on a technology for recognizing a specific object shown in an image. Some image recognition techniques recognize a target in each of the images sequentially acquired over time and track the target.

For example, the following Patent Document 1 describes an image recognition device that detects a person from a camera image and tracks the detected person. This image recognition device detects a person as a detection target from images sequentially acquired with the passage of time, and tracks the person by obtaining the position of the person in each image. In the process of detecting a person, the likelihood that a specific area in the image to be processed indicates a person is obtained, and when the likelihood is larger than a predetermined likelihood threshold, the object corresponding to the specific area is obtained. Is done by confirming as a person.

The object recognized on the image actually refers to an image of an object that occupies a specific area on the image, but in the present specification, in order to simplify the expression, the object occupies a specific area on the image. The image of an object is simply expressed as an object on the image.

Japanese Unexamined Patent Publication No. 2016-162232

The process of recognizing a target object on an image has the following problems. That is, when a plurality of objects having similar likelihoods are recognized from one image, it may be difficult to determine which object should be recognized as the target object. As a result, for example, when tracking a target object in images that are sequentially acquired over time, another object that is not the target object may be mistakenly tracked during the tracking process.

An object of the present invention is to reliably recognize a target shown in an image.

The present invention includes an image acquisition unit that sequentially acquires images with the passage of time, and an object recognition unit that recognizes an object on the image, and the object recognition unit is the object recognized by the previously acquired image. The prediction range setting unit that sets the prediction range in which the target is predicted to be recognized based on the position of the target on the next acquired image, and the object on the next acquired image are described above. It is provided with a target object recognition unit that obtains a likelihood with respect to a target object and recognizes an object satisfying a predetermined position condition with respect to the prediction range as the target object among objects whose likelihood condition satisfies a predetermined recognition range. When there is no object whose likelihood satisfies the recognition condition and the predetermined position condition with respect to the prediction range is not satisfied, the target object recognition unit is an object whose likelihood does not satisfy the recognition condition. When there is a provisional target object that satisfies the predetermined position condition with respect to the prediction range, the target object is recognized by satisfying the recognition condition with the likelihood of the previously acquired images. The correction process of modifying the likelihood of the provisional target according to the number of target recognition images to obtain the correction likelihood, and the provisional target when the correction likelihood satisfies the recognition condition. It is characterized by executing the correction recognition process for recognizing the target object.

Desirably, the correction likelihood has a larger value as the number of target recognition images increases.

According to the present invention, the target object shown in the image can be reliably recognized.

It is a figure which shows the structure of the object recognition apparatus. It is a figure which conceptually shows the process of setting a prediction range. It is a figure which conceptually shows the process of setting a prediction range. It is a figure which shows the example of the output image. It is a flowchart which shows the process of recognizing a target object from one image. It is a figure which shows the example when the likelihood of the object O recognized on the image shown by the 4th image data is not more than the likelihood threshold. It is a figure for demonstrating the tracking error.

FIG. 1 shows the configuration of the object recognition device 10 according to the embodiment of the present invention. The object recognition device 10 includes an image acquisition unit 12, an object recognition unit 14, a tracking result storage unit 26, a program storage unit 28, an object recognition database 30, and a tracking success count storage unit 32. The tracking result storage unit 26, the program storage unit 28, and the tracking success count storage unit 32 may be configured by one memory divided into a plurality of parts by an address.

The object recognition device 10 captures a moving image by the image acquisition unit 12, recognizes and tracks a target object on the moving image. The moving image is composed of individual images sequentially captured by the image acquisition unit 12 over time. That is, the object recognition device 10 recognizes the target object on the images sequentially captured with the passage of time, and tracks the target object by obtaining the position of the recognized target value on each image.

The image acquisition unit 12 includes a camera, acquires (acquires) an image at a predetermined frame rate, and outputs one frame of image data to the object recognition unit 14 every time one frame cycle elapses. The field of view (shooting field of view) of the image indicated by the image data acquired by the image acquisition unit 12 is constant, and the tracking of the target object is performed within the fixed shooting field of view. Further, the image acquisition unit 12 does not have to be provided with a camera, and an image may be acquired from an external camera connected to the image acquisition unit 12. Further, the image acquisition unit 12 may read the moving image data from an external memory, or may read the moving image data from an information processing device that outputs the moving image data.

The object recognition unit 14 includes a prediction range setting unit 18, a target object recognition unit 20, a tracking result generation unit 22, and an output image generation unit 24. The object recognition unit 14 is composed of, for example, a processor that executes a program stored in the program storage unit 28. In this case, the processor executes a program to configure the prediction range setting unit 18, the target recognition unit 20, the tracking result generation unit 22, and the output image generation unit 24 within itself. Further, these components may be individually configured by a digital circuit as hardware.

The object recognition unit 14 searches for a target object in the image indicated by the image data output from the image acquisition unit 12, and tracks the target object in the shooting field of view. Specifically, the target tracking process described below is executed, the position of the target in the image of one frame is obtained, the tracking result data representing the position of the target is obtained for each frame, and the tracking result storage unit 26 To memorize. The tracking result data may be numerical data in which the position of the target object is represented by coordinate values, or may be image data in which the position of the target object is represented by an image. The image showing the position of the target object may be an image in which the image showing the target object is superimposed on the image indicated by the image data acquired by the image acquisition unit 12.

The target tracking process will be described. The prediction range setting unit 18 sets a prediction range in which the target object is recognized with respect to the image indicated by the image data output from the image acquisition unit 12. The prediction range is set using the tracking result data obtained for each of the image data of the M frame acquired earlier than the latest image data. However, when M is an integer of 1 or more and the image data acquired earlier than the latest image data is one frame, the prediction range setting unit 18 is on the image indicated by the image data acquired earlier. The range deviated from the position of the target object recognized in step 1 by a predetermined distance in a predetermined direction is set as the prediction range. Further, when the image data is not acquired before the latest image data, the prediction range setting unit 18 sets a predetermined initial range as the prediction range.

The prediction range setting unit 18 refers to each tracking result data stored in the tracking result storage unit 26, that is, each tracking result data obtained for the image data of the past M frames. Then, the image indicated by the image data finally acquired by the process of obtaining the locus indicating the path of the target moving in the shooting field of view during the M frame and extending the locus to the future side, that is, the latest image Set the prediction range for the latest image shown by the data.

FIG. 2 conceptually shows the process of setting the prediction range. In this figure, a wagon vehicle is shown in the field of view 34 as a target. The targets T1 to T3 are targets recognized on the image indicated by the three-frame image data acquired before the latest image data. The target object T1 is a target object recognized on the image indicated by the first image data acquired first, and the target object T2 is recognized on the image indicated by the second image data acquired next. It is a targeted object. The target object T3 is a target object recognized on the image indicated by the third image data acquired next to the second image data. In FIG. 2, for convenience of explanation, the distance that the target moves in one frame cycle, that is, the distance of the adjacent target is exaggerated. In an actual one-frame cycle, the distance that the target moves in one frame cycle is often shorter than that shown in FIG.

The prediction range setting unit 18 obtains the movement locus of the targets T1 to T3, and sets the prediction range 36 on an extension line of the movement locus toward the future side. In FIG. 2, the rectangular prediction range 36 is indicated by the alternate long and short dash line. The shape of the prediction range 36 may be a circle, an ellipse, or another polygon in addition to a rectangle. Further, the shape of the prediction range 36 may be a shape close to the target object. In addition, in FIG. 2, the position where the wagon vehicle as a target may be searched for on the latest image is shown by a broken line representing the wagon vehicle.

The target object recognition unit 20 detects a region on the latest image whose likelihood is not 0 with respect to the target, recognizes the region as an object, and updates an object whose likelihood with respect to the target exceeds the likelihood threshold H. Search in the image. Here, the likelihood with respect to the target means a value indicating the high possibility that the object is the target (a value indicating the degree of the possibility that the object is the target). In the following description, the "likelihood with respect to the target" is simply referred to as the "likelihood". Likelihood is defined to exceed the likelihood threshold H for an object that is likely to be a target, even for images when the object is viewed from multiple positions with different distances or directions from the object. ..

The process of obtaining the likelihood on the latest image is executed with reference to the object recognition database 30. In the object recognition database 30, multidimensional object data showing the shape when one object is viewed from a plurality of different positions is stored in advance for each of the plurality of types of targets. The plurality of types of objects in which the multidimensional object data is stored may be a plurality of types of objects that the object recognition device 10 intends to recognize, such as an automobile, a motorcycle, a person, and an animal. The target object recognition unit 20 refers to the multidimensional object data of the wagon vehicle as the target object, and searches for an object whose likelihood exceeds the likelihood threshold H in the latest image.

When one object whose likelihood exceeds the likelihood threshold value H is searched on the latest image, the target object recognition unit 20 obtains a determination distance between this object and the prediction range 36. The determination distance is defined as, for example, the distance between the center of gravity of the searched object and the center of gravity of the prediction range 36. When the determination distance is equal to or less than a predetermined distance threshold value, the target object recognition unit 20 recognizes this object as a target object.

When a plurality of objects whose likelihood exceeds the likelihood threshold value H are searched for on the latest image, the target object recognition unit 20 obtains a determination distance for each of the plurality of objects. Among these plurality of objects, the target object recognition unit 20 recognizes the object whose determination distance is equal to or less than a predetermined distance threshold value and whose determination distance is the shortest as the target object.

The tracking result generation unit 22 generates tracking result data in response to the target object recognition unit 20 recognizing the target object, and stores the tracking result data in the tracking result storage unit 26. The tracking result generation unit 22 increases the number of successful tracking times stored in the tracking success count storage unit 32 by one. The number of successful traces is an integer of 0 or more indicating the number of successful traces in which an object whose likelihood exceeds the likelihood threshold H and whose determination distance is equal to or less than the distance threshold is searched. The correction process when the likelihood exceeds the likelihood threshold H and the object whose determination distance is equal to or less than the distance threshold is not searched on the latest image will be described later.

In FIG. 3, in addition to the targets T1 to T3, the target T4 recognized on the image indicated by the fourth image data acquired next to the third image data is shown. By the same processing as the processing executed for the first to fourth image data, the object recognition unit 14 sets the prediction range 36 on the latest image indicated by the latest image data (fifth image data). , An object whose likelihood exceeds the likelihood threshold H is searched for on the latest image. The object recognition unit 14 obtains a determination distance for the searched object, and if the determination distance is equal to or less than the distance threshold value, recognizes this object as a target object. Further, when a plurality of objects whose likelihood exceeds the likelihood threshold value H are searched for in the latest image, the object recognition unit 14 targets the object whose determination distance is equal to or less than the distance threshold value and whose determination distance is the shortest. Recognize that. The object recognition unit 14 further generates tracking result data for the target object and stores it in the tracking result storage unit 26, and increases the number of successful tracking times stored in the tracking success count storage unit 32 by one.

The output image generation unit 24 generates output image data indicating the trajectory of the target by referring to the tracking result data stored in the tracking result storage unit 26, and outputs the output image data from the object recognition unit 14. The output image indicated by the output image data is, for example, an overlay of a figure showing a trajectory of a target object on the latest image. FIG. 4 shows an example of an output image. In the output image 40, the locus 38 of the target object T represented by the curve and the arrow is superimposed on the latest image. The output image generation unit 24 sequentially outputs the output image data corresponding to each image data sequentially output from the image acquisition unit 12 from the object recognition unit 14. Further, the output image generation unit 24 may store each output image data as moving image data in the tracking result storage unit 26, or may store it in another memory connected to the object recognition unit 14.

A correction process will be described when an object whose likelihood exceeds the likelihood threshold H and whose determination distance is equal to or less than the distance threshold is not searched on the latest image. The target object recognition unit 20 searches the latest image for an object whose likelihood exceeds the likelihood threshold H0 and is equal to or less than the likelihood threshold H0.

When the target object recognition unit 20 searches for one object whose likelihood exceeds the likelihood threshold H0 and is equal to or less than the likelihood threshold H on the latest image, the target object recognition unit 20 determines the determination distance between this object and the prediction range 36. Ask for. When the determination distance is equal to or less than a predetermined distance threshold value, the target object recognition unit 20 recognizes this object as a temporary target object.

When a plurality of objects whose likelihood with respect to the target exceeds the likelihood threshold value H0 and are equal to or less than the likelihood threshold value H are searched on the latest image, the target object recognition unit 20 determines the determination distance for each of the plurality of objects. Ask for. Among these plurality of objects, the target object recognition unit 20 recognizes the object whose determination distance is equal to or less than a predetermined distance threshold value and whose determination distance is the shortest as a provisional target object. When the determination distance obtained for each object exceeds a predetermined distance threshold value, the object recognition unit 14 ends the tracking of the target object. In this case, the object recognition unit 14 executes the process of searching for and tracking the target object from the beginning.

The target recognition unit 20 reads the tracking success count SC stored in the tracking success count storage unit 32, and uses the likelihood L0 obtained for the provisional target and the tracking success count SC as follows (Equation 1). ), The likelihood L0 is modified to obtain the modified likelihood L1.

(Equation 1) L1 = L0 + k · SC

k is a positive adjustment factor and is set to an appropriate value by experiment or simulation, for example. In (Equation 1), the larger the tracking success count SC obtained for the image data acquired earlier than the latest image data, the larger the addition value k · SC with respect to the likelihood L0, and the tracking success count SC. The larger the value, the larger the correction likelihood L1.

When the modified likelihood L1 exceeds the likelihood threshold H, the target recognition unit 20 recognizes the provisional target as a target. The tracking result generation unit 22 generates tracking result data in response to the target object recognition unit 20 recognizing the provisional target as the target, and stores the tracking result data in the tracking result storage unit 26.

In the correction process, the number of successful tracking times stored in the tracking success number storage unit 32 is maintained as it is. When the modified likelihood L1 is equal to or less than the likelihood threshold value H, the target recognition unit 20 ends the tracking of the target that has been tracked so far. In this case, the object recognition unit 14 executes the process of searching for and tracking the target object from the beginning.

FIG. 5 shows a flowchart illustrating a process of recognizing a target object on an image in the target object tracking process. The object recognition unit 14 searches for an object by detecting a region having a non-zero likelihood on the image indicated by the image data output from the image acquisition unit 12 (S101). The object recognition unit 14 determines whether or not there is an object whose likelihood exceeds the likelihood threshold value H and whose determination distance is equal to or less than the distance threshold value (S102), and when there are a plurality of such objects, the determination distance is the shortest. (S103). Further, when the likelihood exceeds the likelihood threshold value H and the determination distance is equal to or less than the distance threshold value, the object recognition unit 14 recognizes the object as a target object (S103). The object recognition unit 14 adds 1 to the number of successful tracking times stored in the storage number of successful tracking units 32 (S104). The object recognition unit 14 ends the process of searching for the target object on the image to be processed, and proceeds to the process of recognizing the target object on the next image.

When the likelihood exceeds the likelihood threshold H and there is no object whose determination distance is equal to or less than the distance threshold value, the object recognition unit 14 has a likelihood exceeding the likelihood threshold H0 and equal to or less than the likelihood threshold H. It is determined whether or not there is an object whose determination distance is equal to or less than the distance threshold value (S105), and when there are a plurality of such objects, the object having the shortest determination distance is recognized as a temporary target object (S106). When the likelihood exceeds the likelihood threshold H0, is equal to or less than the likelihood threshold H, and the determination distance is equal to or less than the distance threshold, the object recognition unit 14 uses that object as a tentative target. Recognize (S106).

The object recognition unit 14 modifies the likelihood L0 of the provisional target to obtain the modified likelihood L1 (S107). Then, it is determined whether or not the modified likelihood L1 exceeds the likelihood threshold H (S108). When the modified likelihood L1 exceeds the likelihood threshold H, the object recognition unit 14 recognizes the provisional target as a target (S109), ends the process of searching for the target on the image to be processed, and ends the process. Proceed to the process of recognizing the target on the next image.

When the likelihood exceeds the likelihood threshold H0, is equal to or less than the likelihood threshold H, and there is no object whose determination distance is equal to or less than the distance threshold (S105), the object recognition unit 14 has the modified likelihood of the provisional target. When is equal to or less than the likelihood threshold value H (S108), the process of searching for the target object on the image to be processed is terminated, and the tracking of the target object is terminated.

FIG. 6 shows an example when the likelihood of the object O recognized on the image shown by the fourth image data is equal to or less than the likelihood threshold value H. In this example, the correction likelihood L1 is obtained for the object O, and the object O is recognized as the target object T4 because the correction likelihood L1 exceeds the likelihood threshold value H. The reason why the likelihood becomes equal to or less than the likelihood threshold value is that, for example, the object O to be processed is in the dark, or the likelihood is lowered because the image is taken against the sun. The object recognition unit 14 sets a prediction range on the image indicated by the fifth image data based on the target object T4 in addition to the target objects T1 to T3, continuously executes the target object tracking process, and sets the target object T5. recognize.

As described above, the object recognition device 10 according to the present embodiment includes an image acquisition unit 12 that sequentially acquires images with the passage of time, and an object recognition unit 14 that recognizes a target object on the image. The object recognition unit 14 includes at least a prediction range setting unit 18 and a target object recognition unit 20. The prediction range setting unit 18 sets the prediction range 36 in which the target is predicted to be recognized based on the position of the target recognized in the previously acquired image on the next acquired image. .. The target object recognition unit 20 obtains the likelihood of the next acquired object on the image, and targets an object whose likelihood satisfies a predetermined recognition condition and which satisfies a predetermined position condition with respect to the prediction range 36. Recognize as an object.

The recognition condition in the present embodiment is that the likelihood with respect to the target object exceeds the likelihood threshold value H. Further, the position condition in the present embodiment is a condition that the determination distance between the object or the provisional target and the prediction range 36 is equal to or less than the distance threshold value.

When the target object recognition unit 20 is an object whose likelihood with respect to the target object does not satisfy the recognition condition and there is a provisional target object satisfying a predetermined position condition with respect to the prediction range 36, the target object recognition unit 20 is among the images acquired earlier. , The likelihood of the provisional target is corrected according to the number of successful tracking SCs (the number of target recognition images in which the target is recognized when the likelihood satisfies the recognition condition), and the correction likelihood is obtained. Execute the process. The target object recognition unit 20 executes a correction recognition process for recognizing a provisional target as a target when the correction likelihood satisfies the recognition condition.

Appropriate values may be obtained for the likelihood threshold value H, the likelihood threshold value H0, and the distance threshold value by experiments or simulations.

In the object recognition device 10 according to the present embodiment, an object whose likelihood does not satisfy the recognition condition but which satisfies the position condition with respect to the prediction range is recognized as a provisional target. For the provisional target, the correction likelihood is required so that the value increases as the number of successful trackings in the past increases, and the provisional target can be recognized as the target by the correction likelihood. Therefore, even if the images sequentially acquired by the image acquisition unit 12 deteriorate during the target tracking, the object recognized as the target in the past is easily recognized as the target again, and the target is tracked. Is surely done. It should be noted that the deterioration of the image is due to the fact that the target object enters the region where the field of view is not good or the image is taken against the sun.

In the prior art, when a plurality of objects having similar likelihoods to a target object are in one image, there is a possibility that an object that should not be the target object is erroneously tracked. For example, as shown in FIG. 7, a problem in which a tracking error occurs when a non-target passenger car appears in the shooting field of view 34 in addition to the wagon vehicle as the target will be described. Each of the target objects A1 to A4 conceptually shows a target object recognized from the first to fourth image data sequentially acquired by the image acquisition unit 12 in the photographing field of view 34. The non-target objects B1 to B4 conceptually represent non-target objects recognized from the first to fourth image data in the photographing field of view 34, respectively. It is assumed that the likelihood of the target objects A1, A2 and A4 exceeds the likelihood threshold H, but the likelihood of the target object A3 is equal to or less than the likelihood threshold H.

Since the passenger car has a shape similar to that of a wagon, the likelihood of the non-target objects B1 to B4 is likely to be close to the likelihood of the target objects A1 to A4. Further, the non-target object B3 is close to the target object A3. Under such conditions, in the prior art, the non-target object B3 instead of the target object A3 may be mistakenly recognized as the target object, and thereafter, the passenger car instead of the wagon car may be tracked.

According to the object recognition device 10 according to the present embodiment, there is a high possibility that the non-target object B3 does not satisfy the position condition with respect to the prediction range even if the recognition condition for the likelihood is satisfied. On the other hand, the correction likelihood is obtained for the target object A3, and there is a high possibility that the target object A3 is recognized as the target object based on the correction likelihood. Therefore, the possibility of erroneously recognizing the non-object B3 as the target is reduced, and the target is reliably tracked.

In addition to being defined by (Equation 1), the correction likelihood L1 may be defined so that the larger the number of successful tracking times SC, the larger the value. For example, the correction likelihood L1 may be a value defined by (Equation 2). C in (Equation 2) is an adjustment coefficient exceeding 1, and is set to an appropriate value by experiments and simulations.

(Number 2) L1 = L0 ・ c ・ SC

The object recognition unit 14 may have a learning function for updating the object recognition database 30 by itself. That is, the object recognition unit 14 may have a function of automatically generating multidimensional object data for each classification of targets classified into automobiles, motorcycles, people, and the like. In this case, the object recognition unit 14 may be connected to the Internet. The object recognition unit 14 acquires information representing the appearance characteristics of the object from the Internet for each classification of the target object, generates multidimensional object data, and stores it in the object recognition database 30. When generating the multidimensional object data, the image data acquired by the image acquisition unit 12 may be used as a material.

10 Object recognition device, 12 Image acquisition unit, 14 Object recognition unit, 18 Prediction range setting unit, 20 Target recognition unit, 22 Tracking result generation unit, 24 Output image generation unit, 26 Tracking result storage unit, 28 Program storage unit, 30 Object recognition database, 32 Tracking success count storage, 34 Shooting field, 36 Prediction range, 38 Trajectories, 40 Output images.

Claims (2)

An image acquisition unit that sequentially acquires images over time,
Equipped with an object recognition unit that recognizes the target on the image,
The object recognition unit
A prediction range setting unit that sets a prediction range in which the target is predicted to be recognized based on the position of the target recognized in the previously acquired image on the next acquired image.
Next, the likelihood of the acquired object on the image is obtained with respect to the target object, and among the objects whose likelihood satisfies the predetermined recognition condition, the object satisfying the predetermined position condition with respect to the predicted range is the target object. Equipped with a target recognition unit that recognizes as
When there is no object whose likelihood satisfies the recognition condition and the predetermined position condition with respect to the prediction range is not satisfied, the target object recognition unit may perform the target recognition unit.
When there is an object whose likelihood does not satisfy the recognition condition and there is a provisional target object whose predetermined position condition with respect to the prediction range is satisfied, the likelihood is the recognition condition among the images acquired earlier. The correction process of correcting the likelihood of the provisional target and obtaining the correction likelihood according to the number of target recognition images in which the target is recognized by satisfying the above.
An object recognition device characterized by executing a correction recognition process of recognizing the provisional target as the target when the correction likelihood satisfies the recognition condition. In the object recognition device according to claim 1,
An object recognition device characterized in that the correction likelihood has a larger value as the number of target recognition images increases.

Images