WO2021250934A1

WO2021250934A1 - Image processing device and image processing method

Info

Publication number: WO2021250934A1
Application number: PCT/JP2021/004261
Authority: WO
Inventors: 拓紀茂泉; 宏治土井; 健永崎
Original assignee: 日立Astemo株式会社
Priority date: 2020-06-11
Filing date: 2021-02-05
Publication date: 2021-12-16
Also published as: JPWO2021250934A1; CN115769253A; DE112021002170T5; JP7323716B2

Abstract

Provided are an image processing device and an image processing method with which it is possible to reduce processing load and improve identification accuracy in image processing in which a plurality of identification targets are identified from an image. An image processing device IPA comprises a multi-class identification unit 122, a track processing unit 131, and an identification unit 132. The multi-class identification unit 122 performs multi-class identification processing on an image captured by an imaging device ID to identify multiple types of identification targets. The track processing unit 131 performs image tracking in which each of the identification targets identified in the multi-class identification processing is tracked as a tracking target, to calculate a predicted position of the tracking target in an image at a later point in time, the predicted position being based on an image at an earlier point in time. The identification unit 132 performs two-class identification processing on the predicted position in the image at the later point in time in accordance with the type of tracking target to identify the type of tracking target.

Description

Image processing device and image processing method

This disclosure relates to an image processing apparatus and an image processing method.

Inventions relating to an object detection device, an object detection method, and a program have been conventionally known (see Patent Document 1 below). Patent Document 1 discloses an object detection device including a detection unit and a non-linear processing unit (the same document, abstract, claim 1, paragraph 0006). The detection unit detects one or more object candidate regions from the captured image. The nonlinear processing unit inputs at least a part or all of the captured image including the object candidate region to a neural network that simultaneously estimates the posture of the object in the object candidate region and the distance to the object. Further, the nonlinear processing unit uses the output of the neural network to output object information including at least information on the distance to the object.

The conventional object detection device described in Patent Document 1 detects an object existing in the shooting range based on a shot image taken by an in-vehicle camera, and obtains object information including at least information on the distance to the detected object. Output. Objects detected by the object detection device include other vehicles, pedestrians, two-wheeled vehicles such as bicycles and motorcycles, traffic lights, signs, electric poles, signboards, and other roadside installations that exist around the vehicle on which the object detection device is installed. , Obstacles that may hinder the running of the own vehicle are exemplified (the same document, paragraph 0008).

The detection of the object candidate area by the detection function of the object detection device is based on determining the presence or absence of an object using a scanning rectangle corresponding to the size of the object to be detected on the captured image of the in-vehicle camera. (Ibid., Paragraph 0021). Then, the image feature amount is calculated for the image area in the scanning rectangle, and the discriminator learned in advance is used to determine whether or not there is another vehicle in the scanning rectangle, or the likelihood of indicating the uniqueness of the other vehicle. Is output (ibid., Paragraph 0022).

Japanese Unexamined Patent Publication No. 2019-008660

As the classifier that determines whether or not there is an object to be detected in the scanning rectangle, such as the object detection device, for example, a two-class classifier that discriminates between a vehicle and other objects, a vehicle and a pedestrian, and the like. There are multi-class classifiers that discriminate between multiple objects at once, such as other objects. However, with the development of Advanced Driver Assistance System (ADAS) and Automated Driving System (ADS), the types of identification targets tend to increase further.

In image processing for identifying an object to be identified from an image of an image pickup device, in order to cope with an increase in the types of identification targets, for example, a large number of classifiers may be used together or the hierarchy of each classifier may be increased. Therefore, it is necessary to improve the identification accuracy. However, if the number of classifiers or the hierarchy of classifiers is increased, the load of object discrimination processing increases and the processing time may not be within the required time.

The present disclosure provides an image processing apparatus and an image processing method capable of reducing the processing load of image processing for identifying a plurality of identification targets from an image and improving the identification accuracy.

One aspect of the present disclosure is a multi-class identification unit that performs multi-class identification processing on an image captured by an image pickup device to identify a plurality of types of identification targets, and the identification identified by the multi-class identification process. With respect to the tracking processing unit that performs image tracking with the target as the tracking target and calculates the predicted position of the tracking target in the image at the later time based on the image at the previous time, and the predicted position of the image at the later time. The image processing apparatus is characterized by having an identification unit for identifying the type of the tracking target by performing two-class identification processing according to the type of the tracking target.

According to the above aspect of the present disclosure, it is possible to provide an image processing apparatus and an image processing method capable of reducing the processing load of image processing for identifying a plurality of identification targets from an image and improving the identification accuracy.

The block diagram which shows one Embodiment of the image processing apparatus which concerns on this disclosure. The flow diagram which shows one Embodiment of the image processing method which concerns on this disclosure. The flow diagram which shows one Embodiment of the image processing method which concerns on this disclosure.

Hereinafter, embodiments of the image processing apparatus and the image processing method of the present disclosure will be described with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of the image processing apparatus according to the present disclosure. The image processing device IPA of the present embodiment is, for example, a device that identifies a plurality of types of identification targets from images taken by an image pickup device ID. More specifically, the image processing device IPA is a device mounted on a vehicle and discriminating a plurality of different objects around the vehicle from images taken by an image pickup device ID such as a monocular camera or a stereo camera. .. The image captured by the image pickup apparatus ID is not particularly limited, and for example, a color image or a shade image can be appropriately selected.

In the example shown in FIG. 1, the image pickup device ID is a stereo camera mounted on a vehicle. The image processing device IPA is, for example, a processing unit 100 including a processing device such as a CPU, a storage unit 200 including a storage device such as a ROM or RAM, and a computer stored in the storage unit 200 and executed by the processing unit 100.・ It has a program. Further, although not shown, the image processing apparatus IPA includes, for example, an input / output unit for inputting / outputting signals.

The processing unit 100 of the image processing apparatus IPA has, for example, a signal processing unit 110 and a recognition processing unit 150. The signal processing unit 110 includes, for example, an image acquisition unit 111 and a parallax calculation unit 112. The recognition processing unit 150 includes, for example, a first recognition processing unit 120, a second recognition processing unit 130, and an output processing unit 140. The first recognition processing unit 120 includes, for example, an image area selection unit 121 and a multi-class identification unit 122. The second recognition processing unit 130 includes, for example, a tracking processing unit 131 and an identification unit 132. The identification unit 132 includes, for example, a plurality of two-

class identification units

132a and 132b.

Each unit of the processing unit 100 is, for example, a functional block of the processing unit 100 realized by executing the computer program stored in the storage unit 200 by the processing unit 100. Each of these processing units 100 may be realized by, for example, a dedicated processing device, or a plurality of functional blocks may be realized by one processing device. Further, the storage unit 200 may be configured by, for example, a plurality of storage devices of one type or various types, or may be configured by one storage device.

In the example shown in FIG. 1, the image processing device IPA includes a storage unit 200, but the image processing device IPA may be connected to an external storage unit 200. Further, in the example shown in FIG. 1, the image processing device IPA is connected to the external image pickup device ID, but the image processing device IPA may include the image pickup device ID. Further, in the example shown in FIG. 1, the identification unit 132 has two two-

class identification units

132a and 132b, but may have three or more two-class identification units.

The identification target 202 identified by the image processing device IPA from the image captured by the image pickup device ID is stored in, for example, in the storage unit 200 in advance. The identification target 202 includes a plurality of types such as other vehicles, pedestrians, moving objects, obstacles, roads, road signs, road signs, and signals around the own vehicle equipped with the image processing device IPA. Further, the other vehicles to be identified by the image processing device IPA may include a plurality of types such as light vehicles such as bicycles, motorized bicycles, motorcycles, light vehicles, ordinary vehicles, large vehicles, buses, and trucks. good. Further, the other vehicle may include a type based on the position, attitude, direction of travel, speed, acceleration, angular velocity, etc. with respect to the own vehicle, such as a preceding vehicle, a following vehicle, an oncoming vehicle, a crossing vehicle, a right turn vehicle, and a left turn vehicle. good.

Next, with reference to FIGS. 2A and 2B, an embodiment of the image processing method according to the present disclosure will be described together with the operation of the image processing apparatus IPA shown in FIG. 2A and 2B are flow charts of an image processing method IPM of the present embodiment using the image processing apparatus IPA shown in FIG.

The image pickup device ID captures an image in a predetermined cycle and a predetermined imaging time, for example. The image processing apparatus IPA processes each image taken at a predetermined cycle by the image pickup apparatus ID by the image processing method IPM shown in FIG. 2A. When the image processing apparatus IPA starts the image processing method IPM shown in FIG. 2A, the image processing apparatus IPA first executes the image acquisition process P1.

In the image acquisition process P1, the image acquisition unit 111 acquires, for example, an image from the image pickup apparatus ID and stores it in the storage unit 200 as a part of the image information 201. The image information 201 includes, for example, image information of a right image taken by the right camera and a left image taken by the left camera when the image pickup device ID is a stereo camera.

Further, in the image acquisition process P1, the parallax calculation unit 112 inputs, for example, the right image and the left image, searches for a region in the left image similar to a specific region in the right image, and performs a process of obtaining the parallax. .. The parallax calculation unit 112 outputs a parallax image by performing this process on the entire area of the right image. The parallax calculation unit 112 stores the parallax image in the storage unit 200 as a part of the image information 201.

Next, the image processing device IPA executes, for example, the image area selection process P2. In the image region selection process P2, the image processing apparatus IPA selects an image region that may include any of a plurality of types of identification targets from the images captured by the image pickup apparatus ID. More specifically, the first recognition processing unit 120 acquires, for example, a parallax image which is an output of the parallax calculation unit 112 from the image information 201 stored in the parallax calculation unit 112 or the storage unit 200.

In the image area selection process P2, the image area selection unit 121 receives, for example, a parallax image and groups parallax adjacent to each other and close to each other in the parallax image to generate a rectangular frame surrounding the grouped parallax. .. Further, the image area selection unit 121 selects a rectangular frame whose vertical and horizontal sizes are equal to or larger than a predetermined size as an image area in which any of a plurality of types of identification targets may be included. ..

The image area selection unit 121 outputs and stores the position information of the selected image area, that is, the coordinates on the parallax image, and the vertical and horizontal sizes of the vertical width and the horizontal width as an image area 203 that may include an identification target. Store in unit 200. Here, for example, when a plurality of image areas are selected from a parallax image, the image area selection unit 121 assigns an identification number N from 1 to n (natural number) to each image area, and the storage unit 200 is assigned an identification number N. It is stored as the image area 203.

The image area selection unit 121 estimates the type of identification target that may be included in the image area of the parallax image surrounded by the rectangular frame by, for example, the aspect ratio of the rectangular frame, and identifies the specific type. Only image areas that may contain objects may be selected. When the image pickup device ID is a monocular camera, the image area selection unit 121 may select an image area that may include any of a plurality of types of identification targets from the images of the monocular camera.

In this case, the image area selection unit 121 may use, for example, the detection result of an object by the millimeter wave radar mounted on the vehicle to select the image area. Further, the image area selection unit 121 specifies, for example, a specific area of the image of the image pickup apparatus ID in advance, and performs raster scan using a window of an arbitrary size for the area, so that a plurality of types can be obtained. You may select an image area that may contain any of the identification targets.

Next, the multi-class identification unit 122 executes the process P3 for setting the identification number N of the image area to be processed for the multi-class identification process to N = 1. Further, the multi-class identification unit 122 performs multi-class identification processing on the image captured by the image pickup apparatus ID to perform multi-class identification processing P4 for identifying a plurality of types of identification targets.

More specifically, in the multi-class identification process P4, the multi-class identification unit 122 identifies a plurality of types of identification targets from the image area selected by the image area selection unit 121, for example. The multi-class identification unit 122 sequentially executes the multi-class identification process P4 for each image area having the identification numbers N selected in the above-mentioned selection process P2 from 1 to n.

The multi-class identification process P4 has, for example, a registration number determination process P4a, a multi-class identification process P4b, a type determination process P4c, a type candidate registration process P4d, P4e, and an increment process P4f. First, in the registration number determination process P4a, the multi-class identification unit 122 determines whether or not the number of registrations of the tracking target in the tracking process P6a, which will be described later, is less than the upper limit.

For example, when the multi-class identification unit 122 determines in the registration number determination process P4a that the number of registrations is not less than the upper limit (NO), that is, the number of registrations has reached the upper limit, the processing after the multi-class identification process P4b Is not executed, and the process proceeds to the next process P5. On the other hand, the multi-class identification unit 122 executes the multi-class identification process P4b when, for example, the registration number determination process P4a determines that the number of registrations is less than the upper limit (YES).

For example, in the multi-class identification process P4b, the multi-class identification unit 122 identifies a plurality of types of identification targets stored in the storage unit 200 as the identification target 202 from the image area. The multi-class identification unit 122 evaluates, for example, the similarity between the image area 203 selected by the image area selection unit 121 and stored in the storage unit 200 and the multi-class identification learning data 204 stored in the storage unit 200. ..

The multi-class identification learning data 204 is, for example, learning data obtained by inputting a large number of images of a car to be identified, an image of a motorcycle, and an image of other objects to perform machine learning. That is, the multi-class identification unit 122 performs multi-class identification processing using the multi-class identification learning data 204 obtained by inputting a plurality of types of identification targets and performing machine learning.

More specifically, in the present embodiment, the multi-class identification unit 122 is a machine in which, for example, at least an automobile as an identification target of the first type and a motorcycle as an identification target of the second type are input to the machine. Multi-class identification processing is performed using the learned multi-class identification learning data 204. In the present embodiment, a case where the types of identification targets in the multi-class identification process P4b are, for example, two types of an automobile and a motorcycle will be described, but the types of identification targets and the number thereof are not particularly limited.

The multi-class identification unit 122 calculates, for example, an evaluation value of similarity between the image area 203 and the multi-class identification learning data 204 in the multi-class identification process P4b. Specifically, the multi-class identification unit 122 includes, for example, the multi-class identification learning data 204 in which the identification target (i) of the first type is an automobile and the identification target (ii) of the second type is a motorcycle. , The evaluation value of the similarity with the image area 203 is calculated. Next, the multi-class identification unit 122 executes the determination process P4c of the type of identification target existing in the image area 203 based on the evaluation value of the similarity.

In this determination process P4c, the multi-class identification unit 122 identifies the type of identification target existing in the image area 203, for example, when the evaluation value of the similarity is equal to or higher than a predetermined threshold value. More specifically, in the multi-class identification unit 122, for example, when the evaluation value of the similarity between the image area 203 and the automobile which is the identification target (i) of the first type is equal to or higher than a predetermined threshold value. From the image area 203, the vehicle and its position information, which is the identification target (i) of the first type, are identified. Further, the multi-class identification unit 122 assigns a registration number to the automobile which is the identification target (i) of the first type identified from the image area 203, and registers it in the storage unit 200 as a tracking target and a type candidate 205. The process P4d is executed.

Further, in the determination process P4c, in the multi-class identification unit 122, for example, the evaluation value of the similarity between the image area 203 and the motorcycle which is the identification target (ii) of the second type is equal to or higher than a predetermined threshold value. If this is the case, the image area 203 identifies the motorcycle that is the second type of identification target (ii) and its position information. Further, the multi-class identification unit 122 assigns a registration number to the motorcycle which is the identification target (ii) of the second type identified from the image area 203, and registers the motorcycle as the tracking target and the type candidate 205 in the storage unit 200. Process P4e is executed. The multi-class identification unit 122 executes the increment process P4f after the end of the process P4d or the process P4e, for example.

Further, in the determination process P4c, the multi-class identification unit 122 determines, for example, an evaluation value of similarity between the image area 203 and the identification targets (i) and (ii) of the first and second types. When it is less than the threshold value, it is identified that the image area 203 does not include the identification target. In this case, the multi-class identification unit 122 executes, for example, the increment process P4f.

In the increment process P4f, the multi-class identification unit 122 increments, for example, the identification number N of the image area 203 to be processed by the next multi-class identification process P4 to N + 1. The multi-class identification unit 122 includes, for example, the above-mentioned processes P4a to P4f until the incremented identification number N of the image area 203 exceeds the number n of the image areas 203 selected in the above-mentioned selection process P2. The multi-class identification process P4 is repeatedly executed.

After the completion of the multi-class identification process P4 for all the image areas 203 selected in the above-mentioned selection process P2, the tracking process unit 131 sets the registration number R of the tracking target to be processed in the identification process P6 described later to 1, for example. The process P5 set to is executed. Further, the tracking processing unit 131 calculates the predicted position of the tracking target and the tracking target registered as the type candidate 205 in the storage unit 200, and executes the identification process P6 for determining the type of the tracking target.

The tracking processing unit 131 is, for example, for each tracking target whose registration number R registered as the tracking target and the type candidate 205 in the storage unit 200 in the above-mentioned type candidate registration processes P4d and P4e is 1 to m (natural number). Then, the identification process P6 is sequentially executed. The identification process P6 is, for example, a tracking process P6a, a type candidate determination process P6b, a two-class identification process P6c, P6h, a type determination process P6d, P6i, a registration process P6e, P6j, and a predicted position calculation process P6f, P6k. It has a registration deletion process P6g and an increment process P6l.

First, in the tracking process P6a, the tracking processing unit 131 performs image tracking targeting the identification target identified by the multi-class identification process P4, and predicts the tracking target in the image at the later time based on the image at the previous time. Calculate the position. The tracking processing unit 131 calculates, for example, the predicted position of the tracking target at the current time based on the position information of the tracking target at the previous time.

In the tracking process P6a, the tracking processing unit 131 optically searches for the tracking target at the current time by template matching using the image of the tracking target at the previous time as a template, and the movement amount of each pixel in the tracking target area. A method of estimating by flow etc. is used. Then, the tracking processing unit 131 predicts the movement of the tracking target at the current time from the position of the tracking target at the previous time and the movement amount of the past tracking target.

Further, the tracking processing unit 131 refers to the tracking target and the type candidate 205 registered in the storage unit 200 in, for example, the type candidate determination processing P6b, and the type of the tracking target identified by the multi-class identification processing P4 is the first. It is determined whether the vehicle is of the first type (i) or the motorcycle of the second type (ii).

When the tracking processing unit 131 determines in the type candidate determination processing P6b that the type of the tracking target is a vehicle of the first type (i), the identification unit 132 determines that the predicted position of the image at a later time is reached. Two-class identification processing according to the type of the tracking target is performed to identify the type of the tracking target. More specifically, the identification unit 132 performs the two-class identification process P6c using the two-class identification unit 132a corresponding to the identification target of the first type (i).

The two-class identification unit 132a uses the two-class identification learning data 206 stored in the storage unit 200 to perform two-class identification processing on the predicted position of the identification target and its surroundings to identify the type of the tracking target. Here, the two-class identification learning data 206 is the identification target (i) of the first type, which is one of the identification targets of the plurality of types of the image processing device IPA, that is, the automobile and the other identification targets. This is two-class identification learning data for automobiles in which a large number of images are input and machine learning is performed.

In the two-class identification process P6c, the two-class identification unit 132a calculates, for example, an evaluation value of similarity between the predicted position of the tracking target and the image area around it and the two-class identification learning data 206. Next, the second class identification unit 132a executes the type determination process P6d. In the type determination process P6d, in the second class identification unit 132a, for example, when the evaluation value of the similarity is equal to or higher than a predetermined threshold value, the type of the tracking target is the identification target (i) of the first type. It is determined that the vehicle is a car, and the registration process P6e is executed.

In this registration process P6e, the second class identification unit 132a registers, for example, the vehicle which is the identification target (i) of the first type as the type of the tracking target in the output information 208 of the storage unit 200. Further, in the registration process P6e, the second class identification unit 132a registers the predicted position of the tracking target in the output information 208 of the storage unit 200 as the position of the automobile which is the identification target (i) of the first type. Next, the second class identification unit 132a executes, for example, the predicted position calculation process P6f.

In this predicted position calculation process P6f, the two-class identification unit 132a obtains, for example, the difference between the position information of the tracking target at the previous time and the position information of the tracking target at the current time, and divides the difference by the frame imaging interval. By doing so, the moving speed of the tracking target is calculated. Further, the second class identification unit 132a calculates, for example, the predicted position of the tracking target at the next time based on the position information of the tracking target at the current time and the moving speed of the tracking target. The predicted position of the tracking target calculated here is used, for example, in the tracking process P6a at the next time.

Further, in the above-mentioned type determination process P6d, for example, when the evaluation value of the similarity is less than a predetermined threshold value, the tracking target of the second class identification unit 132a is the identification target (i) of the first type. ) Is determined to be an identification target or background of another type other than the automobile, and the registration deletion process P6g is executed. In the registration deletion process P6g, the two-class identification unit 132a deletes, for example, the tracking target and the type candidate 205 registered in the storage unit 200.

Further, in the above-mentioned type candidate determination processing P6b, when the tracking processing unit 131 determines that the type of the tracking target is a motorcycle of the second type (ii), the identification unit 132 determines that the image at a later time is displayed. The predicted position is subjected to two-class identification processing according to the type of the tracking target to identify the type of the tracking target. More specifically, the identification unit 132 performs the two-class identification process P6h using the two-class identification unit 132b corresponding to the second type (ii).

The two-class identification unit 132b uses the two-class identification learning data 207 stored in the storage unit 200 to perform two-class identification processing on the predicted position of the identification target and its surroundings to identify the type of the tracking target. Here, the two-class identification learning data 207 is the identification target (ii) of the second type, which is one of the identification targets of the plurality of types of the image processing device IPA, that is, the motorcycle and the other identification targets. This is two-class identification learning data for a motorcycle that has been machine-learned by inputting a large number of images of. The two-class identification learning data 206 and the two-class identification learning data 207 may be learned by different machine learning methods.

In the two-class identification process P6h, the two-class identification unit 132b calculates, for example, an evaluation value of similarity between the predicted position of the tracking target and the image area around it and the two-class identification learning data 207. Next, the second class identification unit 132b executes the type determination process P6i. In the type determination process P6i, in the second class identification unit 132b, for example, when the evaluation value of the similarity is equal to or higher than a predetermined threshold value, the type of the tracking target is the second type identification target (ii), that is, It is determined that the vehicle is a motorcycle, and the registration process P6j is executed.

In this registration process P6j, the second class identification unit 132b registers, for example, the motorcycle that is the identification target of the second type (ii) in the output information 208 of the storage unit 200 as the type of the tracking target. Further, in the registration process P6j, the second class identification unit 132b registers the predicted position of the tracking target in the output information 208 of the storage unit 200 as the position of the motorcycle which is the identification target (ii) of the second type. Next, the second class identification unit 132b executes, for example, the predicted position calculation process P6k.

In this predicted position calculation process P6k, the two-class identification unit 132b obtains, for example, the difference between the position information of the tracking target at the previous time and the position information of the tracking target at the current time, and divides the difference by the frame imaging interval. By doing so, the moving speed of the tracking target is calculated. Further, the second class identification unit 132b calculates, for example, the predicted position of the tracking target at the next time based on the position information of the tracking target at the current time and the moving speed of the tracking target. The predicted position of the tracking target calculated here is used, for example, in the tracking process P6a at the next time.

Further, in the above-mentioned type determination process P6i, for example, when the evaluation value of the similarity is less than a predetermined threshold value, the tracking target of the second class identification unit 132b is the identification target (ii) of the second type. ), It is determined that the object is an identification target or a background of another type other than the motorcycle, and the registration deletion process P6g is executed. In the registration deletion process P6g, the two-class identification unit 132b deletes, for example, the tracking target and the type candidate 205 registered in the storage unit 200.

After the completion of the above-mentioned predicted position calculation processing P6f, P6k or registration deletion processing P6g, the tracking processing unit 131 executes, for example, the increment processing P6l. In the increment processing P6l, the tracking processing unit 131 increments, for example, the registration number R of the tracking target and the type candidate 205 to be processed in the next identification processing P6 to R + 1. The tracking processing unit 131 describes the above-mentioned tracking processing P6a until, for example, the incremented registration number R of the tracking target and the type candidate 205 exceeds several meters of the tracking target and the type candidate 205 registered in the multi-class identification process P4. The identification process P6 including the increment process P6l is repeatedly executed.

After the end of the identification process P6, the predicted position and type of the tracking target are output from the second recognition processing unit 130 or the storage unit 200 to the output processing unit 140 as output information 208 as shown in FIG. The output processing unit 140 outputs the output information 208 to, for example, a vehicle control device constituting ADS, ADAS, or the like, so that the output information 208 is used in the signal generation processing in automatic driving or advanced driving support.

Hereinafter, the operation of the image processing apparatus IPA of the present embodiment and the image processing method IPM using the same will be described.

In recent years, ADAS and ADS using an image pickup device ID such as an in-vehicle camera and an external recognition sensor such as a radar have been attracting attention. In image processing for identifying an object to be identified from an image of an image pickup device ID, in order to cope with an increase in the types of identification targets, for example, a large number of classifiers may be used in combination, or the hierarchy of each classifier may be increased. Therefore, it is necessary to improve the identification accuracy. However, if the number of classifiers or the hierarchy of classifiers is increased, the load of object discrimination processing increases and the processing time may not be within the required time.

As described above, the image processing apparatus IPA of the present embodiment has a multi-class identification unit 122, a tracking processing unit 131, and an identification unit 132. The multi-class identification unit 122 performs the multi-class identification process P4 on the image captured by the image pickup apparatus ID to identify a plurality of types of identification targets. The tracking processing unit 131 performs image tracking for the identification target identified by the multi-class identification processing P4, and calculates the predicted position of the tracking target in the image at the later time based on the image at the previous time. The identification unit 132 performs two-class identification processes P6c and P6h according to the type of the tracking target with respect to the predicted position of the image at a later time to identify the type of the tracking target.

Further, the image processing method IPM of the present embodiment performs the multi-class identification process P4 on the image captured by the image pickup apparatus ID to identify a plurality of types of identification targets. Further, the image processing method IPM performs image tracking using the identification target identified by the multi-class identification process P4 as the tracking target, and calculates the predicted position of the tracking target in the image at the later time based on the image at the previous time.
Then, the image processing method IPM performs two-class identification processes P6c and P6h according to the type of the tracking target with respect to the predicted position of the image at a later time to identify the type of the tracking target.

According to the image processing device IPA and the image processing method IPM of the present embodiment, it is possible to reduce the processing load of image processing for identifying a plurality of identification targets from an image and improve the identification accuracy. More specifically, according to the image processing apparatus IPA and the image processing method IPM of the present embodiment, there is a case where a plurality of identification targets are identified from an image using only a multi-class classifier or only a two-class classifier. In comparison, the processing load of image processing can be reduced.

The reason is that by using the multi-class identification process by the multi-class identification unit 122 and the two-class identification process by the identification unit 132 together, the hierarchy of the multi-class identification process can be improved as compared with the case where only the multi-class identification process is used. This is because the processing load can be reduced by making it shallow. By making the hierarchy of the multi-class identification process shallow in this way, even if the identification accuracy of the multi-class identification process deteriorates, the identification target identified by the multi-class identification process is set as the tracking target, and the tracking target is classified into two classes. By performing the processing, it becomes possible to improve the identification accuracy.

In addition, the identification target identified by the multi-class identification process P4 is set as the tracking target, and the two-class identification process is performed for the predicted position of the tracking target of the image at a later time according to the type of the tracking target. The processing can be performed by limiting the types only to the extremely limited image area. This makes it possible to reduce the processing amount of the two-class identification processing and reduce the processing load, and to discriminate erroneous recognition of the multi-class identification processing to improve the identification accuracy of a plurality of types of identification targets. ..

Further, in the image processing apparatus IPA of the present embodiment, the multi-class identification unit 122 performs the multi-class identification process P4 using the multi-class identification learning data 204 obtained by inputting a plurality of types of identification targets and performing machine learning. .. With this configuration, it is possible to perform multi-class identification processing for identifying a plurality of types of identification targets from an image with high accuracy based on the result of machine learning.

Further, in the image processing apparatus IPA of the present embodiment, the multi-class identification unit 122 is subjected to machine learning by inputting at least the identification target (i) of the first type and the identification target (ii) of the second type. The multi-class discrimination process P4 is performed using the multi-class discrimination learning data 204. With this configuration, it becomes possible to accurately discriminate between the identification target (i) of the first type and the identification target (ii) of the second type from among the identification targets of a plurality of types included in the image. ..

Further, in the image processing apparatus IPA of the present embodiment, the identification target (i) of the first type is an automobile, and the identification target (ii) of the second type is a motorcycle. With this configuration, among the plurality of types of identification targets included in the image, the automobile which is the identification target (i) of the first type and the motorcycle which is the identification target (ii) of the second type are accurately selected. It becomes possible to distinguish well.

Further, in the image processing apparatus IPA of the present embodiment, the identification unit 132 has a plurality of two-

class identification units

132a and 132b. Further, the plurality of two-

class identification units

132a and 132b use the two-class

identification learning data

206 and 207 obtained by inputting the identification target of one of the plurality of types and performing machine learning, respectively, for two classes. Perform identification processing. With this configuration, each of the two-class identification unit 132a and the two-class identification unit 132b can accurately determine whether or not one of the plurality of types is the identification target.

Further, in the image processing apparatus IPA of the present embodiment, the number of the two-

class identification units

132a and 132b is equal to the number of types identified by the multi-class identification unit 122. More specifically, in the image processing apparatus IPA of the present embodiment, the multi-class identification unit 122 identifies two types, the first type identification target (i) and the second type identification target (ii). However, the identification unit 132 has two two-

class identification units

132a and 132b. With this configuration, it is possible to perform two-class identification processing on the identification targets of all the types identified by the multi-class identification unit 122, and improve the identification accuracy of the types of the identification targets.

Further, the image processing apparatus IPA of the present embodiment has an image area selection unit 121 that selects an image area that may include any of a plurality of types of identification targets from the image. Then, the multi-class identification unit 122 identifies a plurality of types of identification targets from the image area selected by the image area selection unit 121. With this configuration, the multi-class identification process by the multi-class identification unit 122 can be performed only on a limited image area, the processing amount of the multi-class identification process can be reduced, and the processing load can be reduced.

As described above, according to the present embodiment, there is provided an image processing device IPA and an image processing method IPM capable of reducing the processing load of image processing for identifying a plurality of identification targets from an image and improving the identification accuracy. Can be done.

Although the embodiments of the image processing apparatus and the image processing method according to the present disclosure have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment and does not deviate from the gist of the present disclosure. Any design changes, etc. within scope are included in this disclosure.

121 Image area selection unit 122 Multi-class identification unit 131 Tracking processing unit 132 Identification unit 132a Two-class identification unit 132b Two-class identification unit 204 Multi-class identification learning data 206 Two-class identification learning data 207 Two-class identification learning data ID Imaging device IPA image Processing device IPM image processing method P4 Multi-class identification processing P6c 2 class identification processing P6h 2 class identification processing

Claims

A multi-class identification unit that performs multi-class identification processing on images taken by an image pickup device to identify multiple types of identification targets, and a multi-class identification unit.
An image tracking unit that performs image tracking for the identification target identified by the multi-class identification process and calculates a predicted position of the tracking target in the image at a later time based on the image at the previous time, and a tracking processing unit.
An identification unit that identifies the type of the tracking target by performing two-class identification processing according to the type of the tracking target for the predicted position of the image at the later time.
An image processing apparatus characterized by having.
The first aspect of claim 1, wherein the multi-class identification unit performs the multi-class identification process using the multi-class identification learning data obtained by inputting a plurality of the identification targets of the above type and performing machine learning. Image processing device.
The multi-class identification unit uses the multi-class identification learning data in which at least the first type of identification target and the second type of identification target are input and machine learning is performed, and the multi-class identification process is performed. 2. The image processing apparatus according to claim 2.
The image processing device according to claim 3, wherein the identification target of the first type is an automobile, and the identification target of the second type is a motorcycle.
The identification unit has a plurality of two-class identification units, and has a plurality of two-class identification units.
The plurality of the two-class identification units perform the two-class identification process using the two-class identification learning data obtained by inputting the identification target of the type of one of the plurality of types and performing machine learning. The image processing apparatus according to claim 1, wherein the image processing apparatus is performed.
The image processing apparatus according to claim 5, wherein the number of the two-class identification units is equal to the number of the types identified by the multi-class identification unit.
It has an image area selection unit that selects an image area that may include any one of the above-mentioned identification targets of the above-mentioned type from the above-mentioned image.
The image processing apparatus according to claim 1, wherein the multi-class identification unit identifies a plurality of the identification targets of the type from the image area selected by the image area selection unit.
Multi-class identification processing is performed on the image taken by the image pickup device to identify multiple types of identification targets.
Image tracking is performed on the identification target identified by the multi-class identification process, and the predicted position of the tracking target in the image at the later time based on the image at the previous time is calculated.
An image processing method comprising performing two-class identification processing according to the type of the tracking target on the predicted position of the image at a later time to identify the type of the tracking target.