WO2014084218A1

WO2014084218A1 - Subject detection device

Info

Publication number: WO2014084218A1
Application number: PCT/JP2013/081808
Authority: WO
Inventors: 八木康史; 槇原靖; 華春生; 宮川恵介; 岩崎瞬
Original assignee: 国立大学法人大阪大学
Priority date: 2012-11-27
Filing date: 2013-11-26
Publication date: 2014-06-05
Also published as: JP2014106685A

Abstract

The present invention relates to a subject detection device. A subject identification unit is an identifier in which a feature data group representing an image feature amount is the input and information indicating whether or not a subject (crossing pedestrian) is present is the output, the identifier being generated by machine learning. The identifier produces a feature data group from an image of at least one subregion (82) (non-masked region (102)) selected according to the type of subject from a plurality of subregions (82) constituting each of the learning sample images used for machine learning, and inputs the feature data group.

Description

Object detection device

The present invention relates to an object detection device that detects an object based on a captured image acquired from an imaging means.

Conventionally, various methods have been proposed in which an image is picked up using an image pickup means such as a camera, and an object is detected based on the obtained picked-up image. For example, a technique for detecting a pedestrian reflected in a captured image using a feature amount (so-called HOG feature amount) related to intensity and gradient of luminance in a local region of the image is described.

This technique is described in “Histograms of Oriented Gradients for Human Detection”, IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. 886-893 (2005).

And, by mounting the above-described imaging means and an identifier capable of identifying an object based on the captured image together with the vehicle (one form of the moving body), the object existing outside the vehicle is detected, or A vehicle periphery monitoring device that monitors the relative positional relationship between the object and the host vehicle is constructed. By introducing this device, the vehicle occupant (especially the driver) is supported.

The above discriminator can discriminate not only pedestrians (human bodies) but also animals, artificial structures, etc. by performing various machine learning. However, when extracting an image area (hereinafter referred to as an identification target area) that includes at least these objects, the identification accuracy may differ depending on the type of the object. This is probably because the shape of the projected image varies depending on the type of the object, and the ratio of capturing the image information of the background portion changes. In other words, the image information of the background portion acts as a disturbance factor (noise information) that lowers the learning / identification accuracy of the object during the learning process or the identification process.

In particular, since this type of device sequentially captures images while the vehicle is running, the scene (background, weather, etc.) of the captured image obtained from the camera can change from moment to moment. Moreover, road surface patterns such as pedestrian crossings and guardrails are often included in images of identification objects (pedestrians and the like) acquired when a vehicle travels on a road surface. For this reason, as a result of causing the classifier to perform machine learning, there is a high possibility that the road surface pattern is erroneously learned as a pedestrian. That is, it is difficult to predict the image feature in the background portion, and there is a problem that the influence as a disturbance factor cannot be ignored.

The present invention has been made to solve the above-described problems, and an object of the present invention is to provide an object detection apparatus that can improve learning / identification accuracy regardless of the type of the object.

An object detection apparatus according to the present invention includes an imaging unit that acquires a captured image, an identification target region extraction unit that extracts an identification target region from the captured image acquired by the imaging unit, and the identification target region extraction Object identifying means for identifying, for each type of the object, whether or not the object exists in the identification target area from the image feature amount extracted in the identification target area by the means, The discriminating means is a discriminator generated using machine learning that receives a feature data group as the image feature amount and outputs the presence / absence information of the object, and the discriminator is used for the machine learning. The feature data group is created and input from images of at least one of the sub-regions selected according to the type of the object among the plurality of sub-regions constituting each learning sample image.

Thus, the discriminator as the object discriminating means is an image of at least one sub-area selected according to the type of the object out of a plurality of sub-areas constituting each learning sample image used for machine learning. Since the feature data group is created and input, image information of sub-regions suitable for the shape of the projected image of the object can be selectively employed for the learning process, The learning accuracy can be improved regardless of the type. Then, by excluding the remaining sub-regions from the learning process, it is possible to prevent over-learning for image information other than the projected image of the object that acts as a disturbance factor during the identification process, and the identification accuracy of the object Can be improved.

Further, it is preferable that the object identifying means identifies whether or not the object exists in the identification object area for each moving direction of the object. Based on the tendency that the image shape on the captured image changes according to the moving direction, the learning / identification accuracy of the object is further improved by identifying each moving direction.

Further, it is preferable that the identification target area extracting unit extracts the identification target area having a size corresponding to a distance from the imaging unit to the target. For example, by making the relative magnitude relationship between the target object and the identification target area constant or substantially constant regardless of the distance, the influence of disturbance factors (image information other than the projected image of the target object) can be suppressed uniformly. As a result, the learning / identification accuracy of the object is further improved.

The image feature amount preferably includes a luminance gradient direction histogram in space. Because the fluctuation in the brightness gradient direction due to the exposure amount of imaging is small, it is possible to accurately grasp the characteristics of the target object, and stable identification accuracy even in outdoor environments where the intensity of ambient light changes from moment to moment Is obtained.

Also, it is preferable that the image feature amount includes a luminance gradient direction histogram in time and space. By considering the luminance gradient direction in time as well as in space, it becomes easy to detect and track an object over a plurality of captured images acquired in time series.

Further, it is preferable that the imaging unit is mounted on a moving body and acquires the captured image by capturing an image while the moving body is moving. Since the scene of the captured image obtained from the imaging means mounted on the moving body changes every moment, it is particularly effective.

According to the object detection device of the present invention, the classifier as the object identification unit is selected according to the type of the object from among a plurality of sub-regions constituting each learning sample image provided for machine learning. Since the feature data group is created and input from the at least one sub-region image, image information of the sub-region suitable for the shape of the projected image of the object is selectively selected for the learning process. The learning accuracy can be improved regardless of the type of the object. Then, by excluding the remaining sub-regions from the learning process, it is possible to prevent over-learning for image information other than the projected image of the object that acts as a disturbance factor during the identification process, and the identification accuracy of the object Can be improved. In particular, since the scene of the captured image obtained from the imaging means mounted on the moving body changes every moment, it is particularly effective.

It is a block diagram which shows the structure of the vehicle periphery monitoring apparatus as a target object detection apparatus which concerns on this embodiment. It is a schematic perspective view of the vehicle carrying the vehicle periphery monitoring apparatus shown in FIG. 3A and 3B are image diagrams showing an example of a captured image acquired by imaging using a camera. It is a flowchart with which it uses for description of the learning process by the discrimination device shown in FIG. It is an image figure which shows an example of the learning sample image containing a crossing pedestrian. It is a schematic explanatory drawing regarding the definition method of each sub area | region. It is a detailed block diagram of the discriminator shown in FIG. FIG. 8A is a schematic diagram showing a typical contour image obtained from a large number of learning sample images including crossing pedestrians. FIG. 8B is a schematic explanatory diagram illustrating a non-mask area and a mask area common to each learning sample image including a crossing pedestrian. FIG. 9A is a schematic diagram showing a typical contour image obtained from a large number of learning sample images including facing pedestrians. FIG. 9B is a schematic explanatory diagram showing a non-mask area and a mask area common to each learning sample image including a face-to-face pedestrian. 2 is a flowchart provided for explaining the operation of the ECU shown in FIG. It is a schematic explanatory drawing showing the positional relationship of a vehicle, a camera, and a human body. It is a schematic explanatory drawing regarding the determination method of an identification object area | region. It is a detailed block diagram of the target object identification part shown in FIG. 14A to 14C are schematic explanatory diagrams regarding a method of calculating the HOG feature amount. FIG. 15A is a schematic explanatory diagram relating to a method for calculating the HOG feature amount. FIG. 15B is a schematic explanatory diagram regarding a method of calculating an STHOG feature amount. It is a schematic diagram for demonstrating the principle of the precision maintenance in the detection process using an STHOG feature-value.

Hereinafter, preferred embodiments of the object detection device according to the present invention will be described with reference to the accompanying drawings.

[Configuration of Vehicle Perimeter Monitoring Device 10]
FIG. 1 is a block diagram illustrating a configuration of a vehicle periphery monitoring device 10 as an object detection device according to the present embodiment. FIG. 2 is a schematic perspective view of the vehicle 12 on which the vehicle periphery monitoring device 10 shown in FIG. 1 is mounted.

As shown in FIGS. 1 and 2, the vehicle periphery monitoring apparatus 10 includes a color camera (hereinafter simply referred to as “camera 14”) that captures a color image (hereinafter referred to as a captured image Im) including a plurality of color channels, and A vehicle speed sensor 16 for detecting the vehicle speed Vs of the vehicle 12, a yaw rate sensor 18 for detecting the yaw rate Yr of the vehicle 12, a brake sensor 20 for detecting the brake pedal operation amount Br by the driver, and the vehicle periphery monitoring device 10 An electronic control device (hereinafter referred to as “ECU 22”) to be controlled, a speaker 24 for issuing an alarm or the like by sound, and a display device 26 for displaying a captured image output from the camera 14 are provided.

The camera 14 is a camera that mainly uses light having a wavelength in the visible light region, and functions as an imaging unit that images the periphery of the vehicle 12. The camera 14 has a characteristic that the output signal level increases as the amount of light reflected on the surface of the subject increases, and the luminance (for example, RGB value) of the image increases. As shown in FIG. 2, the camera 14 is fixedly disposed (mounted) at a substantially central portion of the front bumper portion of the vehicle 12.

The imaging means for imaging the periphery of the vehicle 12 is not limited to the above configuration example (so-called monocular camera), and may be, for example, a compound eye camera (stereo camera). Further, an infrared camera may be used instead of the color camera, or both may be provided. Further, in the case of a monocular camera, another ranging means (radar apparatus) may be provided.

Returning to FIG. 1, the speaker 24 outputs an alarm sound or the like in response to a command from the ECU 22. The speaker 24 is provided on a dashboard (not shown) of the vehicle 12. Alternatively, instead of the speaker 24, an audio output function provided in another device (for example, an audio device or a navigation device) may be used.

The display device 26 (see FIGS. 1 and 2) is a HUD (head-up display) arranged on the front windshield of the vehicle 12 at a position that does not obstruct the driver's front view. The display device 26 is not limited to the HUD, but a display that displays a map or the like of a navigation system mounted on the vehicle 12 or a display (MID; multi-information display) that displays fuel consumption or the like provided in a meter unit or the like is used. can do.

The ECU 22 basically includes an input / output unit 28, a calculation unit 30, a display control unit 32, and a storage unit 34.

Each signal from the camera 14, the vehicle speed sensor 16, the yaw rate sensor 18, and the brake sensor 20 is input to the ECU 22 side via the input / output unit 28. Each signal from the ECU 22 is output to the speaker 24 and the display device 26 via the input / output unit 28. The input / output unit 28 includes an A / D conversion circuit (not shown) that converts an input analog signal into a digital signal.

The calculation unit 30 performs calculations based on the signals from the camera 14, the vehicle speed sensor 16, the yaw rate sensor 18, and the brake sensor 20, and generates signals for the speaker 24 and the display device 26 based on the calculation results. The calculation unit 30 functions as a distance estimation unit 40, an identification target region determination unit 42 (identification target region extraction unit), an object identification unit 44 (subject identification unit), and an object detection unit 46. Here, the target object identification unit 44 is configured by a classifier 50 generated using machine learning that receives a feature data group as an image feature amount and outputs the presence / absence information of the target object.

The function of each unit in the calculation unit 30 is realized by reading and executing a program stored in the storage unit 34. Alternatively, the program may be supplied from the outside via a wireless communication device (mobile phone, smartphone, etc.) not shown.

The display control unit 32 is a control circuit that drives and controls the display device 26. The display control unit 32 drives the display device 26 by outputting a signal used for display control to the display device 26 via the input / output unit 28. Thereby, the display apparatus 26 can display various images (captured image Im, a mark, etc.).

The storage unit 34 is a computer-readable and non-transitory storage medium. The storage unit 34 includes an imaging signal converted into a digital signal, a RAM (Random Access Memory) that stores temporary data used for various arithmetic processes, and a ROM (Read Only Memory) that stores an execution program, a table, a map, or the like. ) Etc.

[Outline of operation of vehicle periphery monitoring apparatus 10]
The vehicle periphery monitoring apparatus 10 according to the present embodiment is basically configured as described above. An outline of the operation of the vehicle periphery monitoring device 10 will be described below.

The ECU 22 converts an analog video signal output from the camera 14 into a digital signal at a predetermined frame clock interval / cycle (for example, 30 frames per second) and temporarily captures it in the storage unit 34.

As shown in FIG. 3A, assume that a captured image Im of the first frame is obtained at an imaging time point T = T1. In the illustrated example, in the captured image Im, a road area (hereinafter simply referred to as “road 60”) on which the vehicle 12 travels, and a plurality of utility pole areas (hereinafter simply referred to as “electric poles”) installed along the road 60 at substantially equal intervals. 62 ”) and a crossing pedestrian area on the road 60 (hereinafter simply referred to as“ crossing pedestrian 64 ”).

As shown in FIG. 3B, it is assumed that a captured image Im of the second frame is obtained at an imaging time point T = T2 (> T1). In the captured image Im of this figure, as in the captured image Im shown in FIG. 3A, a road 60, a plurality of utility poles 62, and a crossing pedestrian 64 exist. However, as the frame interval time elapses, the relative positional relationship between the camera 14 and each object changes every moment. Thereby, even if the first and second frames (captured image Im) are in the same field angle range, each object is imaged in a different form (shape, size, or color).

And ECU22 performs various arithmetic processing with respect to the captured image Im (front image of the vehicle 12) read from the memory | storage part 34. FIG. The ECU 22 (particularly the arithmetic unit 30) comprehensively considers the processing results for the captured image Im, and signals (vehicle speed Vs, yaw rate Yr, and operation amount Br) that indicate the traveling state of the vehicle 12 as necessary. A human body, an animal, or the like existing in front is detected as an object to be monitored (hereinafter referred to as “monitoring object” or simply “object”).

When the calculation unit 30 determines that there is a high possibility that the vehicle 12 is in contact with the monitoring target, the ECU 22 controls each output unit of the vehicle periphery monitoring device 10 in order to call the driver's attention. For example, the ECU 22 outputs an alarm sound (for example, a beeping sound) via the speaker 24 and emphasizes the part of the monitored object in the captured image Im visualized on the display device 26. Display.

[Learning process of discriminator 50]
Next, the learning process by the discriminator 50 shown in FIG. 1 will be described in detail with reference to the flowchart of FIG. As a machine learning method, any algorithm of supervised learning, unsupervised learning, and reinforcement learning may be employed. In the present embodiment, “supervised learning” in which learning is performed based on a data set given in advance will be described with a specific example.

In step S11, learning data used for machine learning is collected. Here, the learning data is a data set of a learning sample image including (or not including) the target object and the type of the target object (including the attribute “no target object”). Examples of types of objects include human bodies, various animals (specifically, mammals such as deer, horses, sheep, dogs and cats, birds, etc.), artificial structures (specifically, vehicles, signs, utility poles) , Guardrails, walls, etc.).

FIG. 5 is an image diagram showing an example of learning

sample images

74 and 76 including crossing

pedestrians

70 and 72. At the approximate center of the learning sample image 74, a projected image of a human body that crosswalks from the left side to the right side (hereinafter referred to as a crossing pedestrian 70) is displayed. In addition, a projected image of a human body that crosswalks from the right front side to the left back side (hereinafter referred to as crossing pedestrian 72) is displayed at the approximate center of another learning sample image 76. Each of the

learning sample images

74 and 76 in this example corresponds to a correct image including the object. Each learning sample image collected may include an incorrect image that does not include an object. Further, in order to unify the data format input to the discriminator 50 (see FIG. 1) side, it is preferable that the

learning sample images

74 and 76 have an image region 80 whose shapes match or are similar to each other.

In Step S12, a plurality of sub-regions 82 are defined by dividing the image region 80 of each of the

learning sample images

74 and 76, respectively. In the example of FIG. 6, the rectangular image region 80 is equally divided into a lattice shape with eight rows and six columns regardless of the size. That is, 48 sub-regions 82 having the same shape are defined in the image region 80, respectively.

In step S13, the learning architecture of the classifier 50 is constructed. Examples of the learning architecture include boosting method, SVM (Support Vector Machine), neural network, EM (Expectation Maximization) algorithm and the like. In this embodiment, AdaBoost, which is a kind of boosting method, is applied.

As shown in FIG. 7, the discriminator 50 includes N (N is a natural number of 2 or more) feature data generators 90, N weak learners 92, a weight updater 93, and a weight calculator 94. And a sample load updater 95.

In this figure, for convenience of explanation, the first data generator, the second data generator, the third data generator,..., The Nth data generator are sequentially arranged from the top with respect to each feature data generator 90. It is written. Similarly, the first weak learner, the second weak learner, the third weak learner,..., And the Nth weak learner are written in order from the top for each weak learner 92.

As shown in the figure, one feature data generator 90 (for example, the first data generator) and one weak learner 92 (for example, the first weak learner) are alternatively connected, N subsystems are constructed. The output side (each weak learner 92) of N subsystems is connected to the input side of the weight updater 93, and the weighting calculator 94 and the sample load updater 95 are connected in series to the output side. It is connected. The detailed operation of the classifier 50 during machine learning will be described later.

In step S14, the mask conditions for the sub-region 82 are determined for each type of object. Here, the mask condition is a selection as to whether or not to adopt an image in each sub-region 82 when N feature data (hereinafter referred to as a feature data group) is generated from one learning sample image 74. Means a condition.

For example, when the captured image Im actually obtained by mounting the camera 14 on the vehicle 12 is used as the learning sample images 74 and 76 (correct images), various structures may be reflected behind the object. In the learning sample image 74 shown in FIG. 5, the background portion 78 excluding the crossing pedestrian 70 shows another three human bodies, a road surface, a building wall, and the like.

In machine learning, if a feature data group including not only the crossing pedestrian 70 but also the background portion 78 is created, over-learning related to the background portion 78 may occur due to the tendency of the database storing the learning sample images 74 and the like. In this case, the image information of the background part 78 acts as a disturbance factor (noise information) that reduces the learning / identification accuracy of the crossing pedestrian 70 as the object. Therefore, it is effective to select only the sub-region 82 suitable for the identification process from among all 48 sub-regions 82 and to learn using the created feature data group.

FIG. 8A is a schematic diagram showing a typical contour image 100 obtained from a large number of

learning sample images

74 and 76 including crossing

pedestrians

70 and 72. Specifically, a contour extraction image (not shown) in which the contour of each object is extracted by performing known edge extraction processing on each learning sample image 74 or the like using an image processing device (not shown). Create each one. Here, each contour extraction image represents a part from which the contour of the object has been extracted in white, and represents a part from which the outline has not been extracted in black.

Then, by using various image processing (for example, averaging processing) for each contour extraction image using an image processing device (not shown), a contour image typical for each learning sample image 74 (typical contour image 100). ) Here, similar to each contour extraction image, the typical contour image 100 represents a part from which the contour of the object has been extracted in white, and represents a part from which the contour has not been extracted in black. That is, the typical contour image 100 corresponds to an image representing the contour of an object (crossing pedestrians 70 and 72) included in common in a large number of learning sample images 74 and the like.

For example, a sub-region 82 whose contour feature amount exceeds a predetermined threshold is adopted as a calculation target, and a sub-region 82 whose contour feature amount falls below a predetermined threshold is excluded from the calculation target. As a result, as shown in FIG. 8B, among the 48 sub-regions 82 defined in the image region 80, a set of 20 sub-regions 82 (region showing a partial image of the typical contour image 100) is obtained. It is assumed that the non-mask area 102 is determined. Of the 48 sub-regions 82, the remaining 28 sub-regions 82 (regions filled in with white) are determined as the mask region 104.

FIG. 9A is a schematic diagram showing a typical contour image 106 obtained from a large number of learning sample images including facing pedestrians. The method for creating the typical contour image 106 is the same as that for the typical contour image 100 in FIG.

FIG. 9B is a schematic explanatory diagram showing a non-mask area 108 and a mask area 110 that are common to each learning sample image including a face-to-face pedestrian. Since the determination method of the mask area 110 is the same as that of the mask area 104 in FIG. 8B, the description thereof is omitted.

As can be understood from FIGS. 8B and 9B, the mask areas 104 and 110 are different even though the object is the same (pedestrian). More specifically, the mask areas 104 and 110 differ depending on whether or not the four sub-areas 82 (see FIG. 9B) with hatching are mask targets. This is because the degree of change in the image shape due to walking motion (movement of shaking hands and feet) varies depending on the moving direction of the pedestrian. In this way, learning may be performed for each moving direction of the object based on the tendency of the image shape on the image to change according to the moving direction. The moving direction may be any of a transverse direction (more specifically, right direction and left direction), a facing direction (more specifically, near side direction and back direction), and an oblique direction with respect to the image plane.

In step S15, machine learning is performed by sequentially inputting a large number of learning data collected in step S11 to the discriminator 50. Hereinafter, it will be described in detail with reference to FIG.

First, the discriminator 50 inputs the learning sample image 74 including the crossing pedestrian 70 among the collected learning data to the feature data generator 90 side. Then, each feature data generator 90 creates each feature data (collectively, feature data group) by performing specific arithmetic processing on the learning sample image 74 according to the mask condition determined in step S14. As a specific method of the mask processing, the calculation may be performed without using the values of all pixels belonging to the mask area 104 (see FIG. 8B), or the values of all the pixels described above are replaced with predetermined values (for example, 0). The feature data may be substantially invalidated by calculating later.

Each weak learner 92 (i-th weak learner; 1 ≦ i ≦ N) is predetermined for each feature data (i-th feature data) acquired from the feature data generator 90 (i-th data generator). Each output result (i-th output result) is obtained by performing the above calculation. The weight updater 93 receives the first to Nth output results acquired from the weak learners 92, and the object information 96 which is the presence / absence information of the object in the collected learning data. Enter. Here, the object information 96 indicates that the crossing pedestrian 70 is included in the learning sample image 74.

Thereafter, the weight updater 93 selects one weak learner 92 that has obtained an output result that minimizes the amount of error from the output value corresponding to the object information 96, and updates the weighting coefficient α so as to increase. The quantity Δα is determined. The weighting calculator 94 updates the weighting coefficient α by adding the update amount Δα supplied from the weight updater 93. At the same time, the sample load updater 95 updates a load (hereinafter referred to as a sample load 97) previously applied to the learning sample image 74 and the like based on the updated weighting coefficient α and the like.

In this way, the learning data is input, the weighting coefficient α is updated, and the sample weight 97 is sequentially updated, and the discriminator 50 performs machine learning until the convergence condition is satisfied (step S15).

As described above, by mounting the machine-learned discriminator 50 in the vehicle periphery monitoring device 10 (see FIG. 1), whether at least one type of object exists in the discrimination target area of the captured image Im. An object identification unit 44 that can identify whether or not is constructed. The object identifying unit 44 may be configured to be able to identify not only the object but also an object excluding the object (for example, the road 60 in FIG. 3A and the like).

[Detailed Operation of Vehicle Perimeter Monitoring Device 10]
Next, the detailed operation of the vehicle periphery monitoring device 10 will be described with reference to the flowchart of FIG. Note that this processing flow is executed for each imaging frame when the vehicle 12 is traveling.

In step S21, the ECU 22 acquires a captured image Im that is an output signal in front of the vehicle 12 (predetermined angle of view range) captured by the camera 14 for each frame. For example, as shown in FIG. 3A, it is assumed that a captured image Im in units of frames is obtained at the time T = T1. Then, the ECU 22 temporarily stores the acquired captured image Im in the storage unit 34. For example, when an RGB camera is used as the camera 14, the obtained captured image Im is a multi-gradation image composed of three color channels.

In step S22, the distance estimation unit 40 estimates the distance Dis from the vehicle 12 by calculating the elevation angle of the vehicle 12 using the captured image Im acquired in step S21.

FIG. 11 is a schematic explanatory diagram showing the positional relationship between the vehicle 12, the camera 14, and the human body M. Here, it is assumed that the vehicle 12 on which the camera 14 is mounted and the human body M as an object are present on a flat road surface S. A contact point between the human body M and the road surface S is Pc, an optical axis of the camera 14 is L1, and a straight line connecting the optical center C of the camera 14 and the contact point Pc is L2.

For example, the angle (elevation angle) formed by the optical axis L1 of the camera 14 with respect to the road surface S is β, the angle formed by the straight line L2 with respect to the optical axis L1 is γ, and the height of the camera 14 with respect to the road surface S is Let it be Hc. In this case, the distance Dis is calculated by the following equation (1) using the angles β, γ, and the height Hc from a geometrical consideration.
Dis = Hc / tan (β + γ) (1)

In this way, the distance estimation unit 40 can estimate the distance Dis corresponding to each position of the road surface S (the road 60 in FIG. 3A) on the captured image Im. Alternatively, the distance estimation unit 40 considers the posture change between the road surface S and the camera 14 due to the motion of the vehicle 12, and estimates the distance Dis using a known method such as SfM (Structure from Motion). Also good. In addition, when the vehicle periphery monitoring apparatus 10 is provided with a distance measuring sensor, you may measure the distance Dis using this.

In step S23, the identification target area determination unit 42 determines the size and the like of the identification target area 122, which is the image area to be identified. In the present embodiment, the identification target area determination unit 42 determines the size or the like of the identification target area 122 according to the distance Dis estimated in step S22 and / or the vehicle speed Vs acquired from the vehicle speed sensor 16. A specific example will be described with reference to FIG.

12, the position on the captured image Im corresponding to the contact point Pc (see FIG. 11) between the road surface S (road 60) and the human body M (crossing pedestrian 64) is defined as a reference position 120. And the rectangular identification object area | region 122 is set so that all the crossing pedestrians 64 may be included.

The identification target area determination unit 42 determines the size of the identification target area 122 according to the distance Dis from the camera 14 (optical center C in FIG. 11) to the target using an arbitrary calculation formula including a linear function and a nonlinear function. decide. When it is assumed that the crossing pedestrian 64f illustrated by a broken line exists at the reference position 124 on the road surface S (the road 60), an identification target area 126 similar to the identification target area 122 is set. For example, by setting the relative magnitude relationship between the crossing pedestrian 64 (64f) and the identification target region 122 (126) to be constant or substantially constant regardless of the distance Dis, a disturbance factor (other than the projected image of the target) The influence of the image information) can be uniformly suppressed, and as a result, the learning / identification accuracy of the object is further improved.

Then, the identification target area determination unit 42 also determines a designated area 128 that is a target range of a raster scan described later. For example, Dis1 that is a distance in which the object is reliably detected within the normal operation range is determined as the lower limit value, and Dis2 that is a distance that does not cause a collision with the object immediately within the normal operation range is set as the upper limit value. May be determined as As described above, by omitting scanning in a part of the captured image Im, not only can the calculation amount and calculation time of the identification process be reduced, but also erroneous detection itself that may occur in a region other than the designated region 128. Can be eliminated.

Note that the identification target region determination unit 42 may appropriately change the shape of the identification target region 122 according to the type of the target object. In this case, similarly to the case of the present embodiment, the size may be determined according to the distance Dis (for example, a value proportional to the distance Dis).

Further, the identification target area determination unit 42 may change the size of the identification target area 122 according to the height of the target object even at the same distance Dis. Thereby, it is possible to set an appropriate size according to the height of the object, and the learning / identification accuracy of the object is further improved by uniformly suppressing the influence of the disturbance factor.

In step S24, the arithmetic unit 30 starts a raster scan of the captured image Im within the designated area 128 determined in step S23. Here, the raster scan refers to a method of successively identifying the presence or absence of an object while moving the reference position 120 (pixels in the captured image Im) in a predetermined direction. Hereinafter, the identification target area determination unit 42 sequentially determines the reference position 120 currently being scanned and the position / size of the identification target area 122 identified from the reference position 120.

In step S25, the object identifying unit 44 identifies whether or not there is at least one kind of object in the determined identification target area 122.

As shown in FIG. 13, the object identifying unit 44 is a classifier 50 (see FIG. 7) generated using machine learning. In the weighting calculator 94, an appropriate weighting coefficient αf obtained by machine learning (step S15 in FIG. 4) is preset.

The object identifying unit 44 inputs an evaluation image 130 having an image area 80 including the identification target area 122 to each feature data generator 90 side. Here, in order to increase the identification accuracy by the object identification unit 44, necessary image processing such as normalization processing (gradation processing / enlargement / reduction processing), alignment processing, etc. may be appropriately performed on the image of the identification target region 122. Good. The object identification unit 44 includes each feature data generator 90, each weak learner 92, a weighting calculator 94, and an integrated learner that applies a step function to the weighted output result acquired from the weighting calculator 94. The evaluation image 130 is sequentially processed via 98, and an identification result indicating that the crossing pedestrian 64 exists in the identification target area 122 is output. In this case, the object discriminating unit 44 functions as a strong discriminator having high discrimination performance by combining N weak discriminators (weak learners 92).

Each feature data generator 90 calculates an image feature amount (that is, the above-described feature data group) in each sub-region 82 using the same calculation method as in the learning process (see FIG. 7).

Incidentally, various known methods can be used as the image feature amount calculation method. Hereinafter, the HOG (Histograms of Oriented Gradient: luminance gradient direction histogram) feature amount indicating the intensity and gradient characteristics of the luminance in the local region of the image will be described.

As shown in FIG. 14A, it is assumed that one block, which is a unit for creating a histogram, is selected from the image area 80. In order to facilitate understanding of the technology, each block is defined below corresponding to each sub-region 82. For example, it is assumed that the sub-region 82 as a block is composed of a total of 36 pixels 84, 6 pixels vertically and 6 pixels horizontally.

As shown in FIG. 14B, a two-dimensional gradient (Ix, Iy) of luminance is calculated for each pixel 84 constituting the block. In this case, the gradient intensity I and the spatial luminance gradient angle θ are calculated according to the following equations (2) and (3).
I = (Ix ² + Iy ² ) ^1/2 (2)
θ = tan ⁻¹ (Iy / Ix) (3)

The arrows written in each grid in the first row illustrate the direction of the planar luminance gradient. Actually, the gradient intensity I and the spatial luminance gradient angle θ are calculated for all the pixels 84, but the illustration of arrows in the second and subsequent rows is omitted.

As shown in FIG. 14C, a histogram for the spatial luminance gradient angle θ is created for each block. The horizontal axis of the histogram is the spatial luminance gradient angle θ (eight divisions in this example), and the vertical axis of the histogram is the gradient intensity I. In this case, a histogram for each block is created based on the gradient intensity I shown in Equation (2).

As shown in FIG. 15A, the HOG feature amount of the evaluation image 130 is obtained by connecting the histogram for each block (spatial luminance gradient angle θ in the example of FIG. 14C) in a predetermined order, for example, ascending order.

Thus, the image feature quantity may include a luminance gradient direction histogram (HOG feature quantity) in space. Because the fluctuation of the luminance gradient direction (θ) due to the exposure amount of imaging is small, it is possible to accurately capture the characteristics of the target object, and it is stable even in outdoor environments where the intensity of ambient light changes from moment to moment Identification accuracy can be obtained.

Further, as the image feature amount, an STHOG (Spatio-Temporal Histograms of Oriented Gradient) feature amount in which the definition of the HOG feature amount is expanded may be used. In this case, a three-dimensional gradient (Ix, Iy, It) of luminance is calculated for each pixel 84 constituting the block using a plurality of captured images Im acquired in time series, as in FIG. 14B. Is done. Here, the gradient intensity I and the temporal luminance gradient angle φ are calculated according to the following equations (4) and (5).
I = (Ix ² + Iy ² + It ² ) ^1/2 (4)
φ = tan ⁻¹ {It / (Ix ² + Iy ² ) ^1/2 } (5)

As shown in FIG. 15B, an STHOG feature quantity which is a brightness gradient histogram in space-time is obtained by further connecting a histogram of the temporal brightness gradient angle φ to the HOG feature quantity. Thus, not only the luminance gradient direction (θ) in space but also the luminance gradient direction (φ) in time is taken into consideration, thereby detecting an object over a plurality of captured images Im acquired in time series.・ Easy tracking.

In the case of the STHOG feature value, a histogram for each block is created based on the gradient intensity I shown in the equation (2), similarly to the HOG feature value.

In this way, the object identification unit 44 identifies whether or not at least one type of object exists in the determined identification object region 122 (step S25). Thereby, not only the kind of the object including the human body M but also the moving direction including the transverse direction and the facing direction are identified.

In step S26, the identification target area determination unit 42 determines whether or not all the scans in the designated area 128 have been completed. When it is determined that it is not completed (step S26: NO), the process proceeds to the next step (S27).

In step S27, the identification target area determination unit 42 changes the position or size of the identification target area 122. Specifically, the identification target area determination unit 42 moves the reference position 120 that is the scan target by a predetermined amount (for example, one pixel) in a predetermined direction (for example, the right direction). When the distance Dis changes, the size of the identification target area 122 is also changed. Furthermore, considering that the typical value of the body length or the body width varies depending on the type of the object, the identification target region determination unit 42 may change the size of the identification target region 122 according to the type of the target.

Thereafter, returning to step S25, the arithmetic unit 30 sequentially repeats steps S25 to S27 until all the scans in the designated area 128 are completed. When it is determined that the processing has been completed (step S26: YES), the calculation unit 30 ends the raster scan of the captured image Im (step S28).

In step S29, the object detection unit 46 detects an object present in the captured image Im. The identification result for a single frame may be used, or the motion vector for the same object can be calculated by considering the identification results for a plurality of frames.

In step S30, the ECU 22 causes the storage unit 34 to store data necessary for the next calculation process. For example, the distance Dis created in step S22, the attribute of the object (crossing pedestrian 64 in FIG. 3A, etc.) obtained in step S25, the reference position 120, and the like can be given.

By sequentially executing this operation, the vehicle periphery monitoring device 10 can monitor an object (for example, the human body M in FIG. 11) existing in front of the vehicle 12 at a predetermined time interval.

[Effect in this embodiment]
As described above, the vehicle periphery monitoring apparatus 10 is extracted with the camera 14 that acquires the captured image Im, and the identification target region determination unit 42 that extracts the

identification target regions

122 and 126 from the acquired captured image Im. The object identification unit for identifying, for each type of object, whether or not an object (for example, a crossing pedestrian 64) exists in the

identification object areas

122 and 126 from the image feature amounts in the

identification object areas

122 and 126. 44. The target object identification unit 44 is a classifier 50 generated using machine learning that receives a feature data group as an image feature amount and outputs the presence / absence information of the target object.

The discriminator 50 as the object discriminating means includes at least one sub selected according to the type of the object out of the plurality of sub areas 82 constituting the

learning sample images

74 and 76 used for machine learning. Since the feature data group is created and input from the image of the region 82 (non-mask regions 102 and 108), the image information of the sub-region 82 suitable for the shape of the projected image of the object is obtained for the learning process. The learning accuracy can be improved regardless of the type of object. Further, by excluding the remaining sub-region 82 (mask regions 104 and 110) from the learning process, it is possible to prevent over-learning with respect to image information other than the projected image of the object that acts as a disturbance factor during the identification process. Yes, the identification accuracy of the object can be improved.

Further, it is preferable that the camera 14 is mounted on the vehicle 12 and acquires the captured image Im by capturing an image while the vehicle 12 is moving. Since the scene (background, weather, road surface pattern, etc.) of the captured image Im obtained from the camera 14 mounted on the vehicle 12 changes every moment, it is particularly effective.

[Relationship with STHOG features]
By the way, if the detection process using the STHOG feature amount is executed under a state where the scene of the captured image Im changes dynamically, the detection accuracy of the target object may be lowered. This is because the detection result of the object can be affected by erroneously detecting the movement component caused by the traveling state of the vehicle 12 as the movement component of the background portion that is substantially stationary.

Therefore, the identification target area determination unit 42 extracts and determines the identification target area 122 based on the position on the road 60 (the reference position 120 in FIG. 12), thereby detecting the same as when the scene is fixed. Accuracy can be maintained. Hereinafter, the principle of maintaining accuracy in the detection process using the STHOG feature amount will be described with reference to FIG.

As illustrated in FIG. 16, time-series data 132 used for calculation of the STHOG feature value is obtained by extracting the identification target region 122 on the xy plane in time series. Here, the reference position 120 included in each identification target region 122 corresponds to a position on the road 60 (FIG. 3A and the like). That is, the identification target area 122 as the vicinity area of the reference position 120 corresponds to a set of points that are completely or substantially stationary.

Thereby, when detecting the object (crossing pedestrian 64), it becomes possible to cancel the motion component of the background portion where the relative speed with respect to the road 60 is substantially zero. As a result, even in a state where the scene changes dynamically, the detection accuracy equivalent to that when the scene is fixed can be maintained.

In addition to or separately from this, the identification target region determination unit 42 extracts and determines the identification target region 122 having a size corresponding to the distance Dis, thereby determining the relative position between the target and the background portion and The time series data 132 having substantially the same size and having a stable image shape can be obtained.

[Supplement]
In addition, this invention is not limited to embodiment mentioned above, Of course, it can change freely in the range which does not deviate from the main point of this invention.

In the present embodiment, the above-described identification process is performed on the captured image Im obtained by the monocular camera (camera 14), but it goes without saying that the same effect can also be obtained with a compound eye camera (stereo camera). .

In this embodiment, the learning process and the identification process by the object identification unit 44 (identifier 50) are performed separately, but both processes may be provided so as to be executed in parallel.

In the present embodiment, the entire vehicle periphery monitoring device 10 is mounted on the vehicle 12, but any configuration may be used as long as at least imaging means is mounted. For example, even in the configuration in which the imaging signal output from the imaging unit is transmitted to a separate arithmetic processing unit (including the ECU 22) via the wireless communication unit, the same effect as that of the present embodiment can be obtained.

In the present embodiment, the object detection device is applied to the vehicle 12, but is not limited to this, and may be applied to other types of moving objects (for example, ships, aircraft, artificial satellites, etc.). Good. Needless to say, even when the object detection device is fixedly arranged, a certain effect of improving the learning / identification accuracy of the object can be obtained.

Claims

Imaging means (14) for acquiring a captured image (Im);
An identification target area extraction means (42) for extracting an identification target area (122, 126) from the captured image (Im) acquired by the imaging means (14);
Whether or not the object (64) exists in the identification target area (122, 126) from the image feature amount in the identification target area (122, 126) extracted by the identification target area extraction means (42). And object identification means (44) for identifying the object for each type of the object (64),
The object identification means (44) is a classifier (50) generated using machine learning, which receives a feature data group as the image feature quantity and outputs presence / absence information of the object (64). Yes,
The discriminator (50) selected according to the type of the object (64) out of a plurality of sub-regions (82) constituting each learning sample image (74, 76) used for the machine learning The object detection device (10), wherein the feature data group is created and input from an image of at least one of the sub-regions (82).
Device (10) according to claim 1,
The object identification means (44) identifies whether or not the object (64) exists in the identification object area (122, 126) for each moving direction of the object (64). An object detection device (10).
Device (10) according to claim 1,
The identification target area extracting unit (42) extracts the identification target area (122, 126) having a size corresponding to a distance from the imaging unit (14) to the target (64). Object detection device (10).
Device (10) according to claim 1,
The object detection device (10), wherein the image feature amount includes a luminance gradient direction histogram in space.
Device (10) according to claim 1,
The object detection device (10), wherein the image feature amount includes a luminance gradient direction histogram in time and space.
The device (10) according to any one of claims 1 to 5,
The imaging means (14) is mounted on the moving body (12) and acquires the captured image (Im) by capturing an image while the moving body (12) is moving. Device (10).