JP2014106685A - Vehicle periphery monitoring device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a vehicle periphery monitoring device capable of improving learning accuracy and identification accuracy regardless of the kind of an object.SOLUTION: An object identification part 44 is a discriminator 50 created using machine learning, into which feature data group as an image characteristic amount is inputted, and from which presence/absence information on objects (road-crossing pedestrians 70, 72) is outputted. The discriminator 50 creates the feature data group from an image of at least one sub-region 82 (non-mask regions 102, 108) selected out of a plurality of sub-regions 82 constituting respective learning sample images 74, 76 supplied to the machine learning in accordance with the kind of the object and inputs the created feature data group thereinto.

Description

  The present invention relates to a vehicle periphery monitoring device that detects an object existing around a vehicle based on a captured image acquired from an on-vehicle imaging unit.

  Conventionally, various methods have been proposed in which an image is picked up using an image pickup means such as a camera, and an object is detected based on the obtained picked-up image. For example, a method for detecting a pedestrian reflected in a captured image using a feature amount (so-called HOG feature amount) related to intensity and gradient of luminance in a local region of the image is described (see Non-Patent Document 1). ).

  Then, by mounting the above-described imaging means and an identifier capable of identifying an object based on the captured image together with the vehicle, the object existing outside the vehicle is detected, or the relative between the object and the host vehicle is detected. A vehicle periphery monitoring device for monitoring a general positional relationship and the like is constructed. By introducing this device, the vehicle occupant (especially the driver) is supported.

“Histograms of Oriented Gradients for Human Detection”, IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. 886-893 (2005)

  The discriminator described above can discriminate not only pedestrians (human bodies) but also animals, artificial structures and the like by performing various machine learning. However, when extracting an image area (hereinafter referred to as an identification target area) that includes at least these objects, the identification accuracy may differ depending on the type of the object. This is probably because the shape of the projected image varies depending on the type of the object, and the ratio of capturing the image information of the background portion changes. In other words, the image information of the background portion acts as a disturbance factor (noise information) that lowers the learning / identification accuracy of the object during the learning process or the identification process.

  In particular, since this type of vehicle periphery monitoring apparatus sequentially captures images while the vehicle is running, the scene (background, weather, etc.) of the captured image obtained from the camera can change from moment to moment. Moreover, road surface patterns such as pedestrian crossings and guardrails are often included in images of identification objects (pedestrians and the like) acquired when a vehicle travels on a road surface. For this reason, as a result of causing the classifier to perform machine learning, there is a high possibility that the road surface pattern is erroneously learned as a pedestrian. That is, it is difficult to predict the image feature in the background portion, and there is a problem that the influence as a disturbance factor cannot be ignored.

  The present invention has been made to solve the above-described problem, and an object thereof is to provide a vehicle periphery monitoring device that can improve learning / identification accuracy regardless of the type of an object.

  A vehicle periphery monitoring device according to the present invention is mounted on a vehicle and captures an image of the captured image acquired by the imaging unit by acquiring an image of the periphery of the vehicle by capturing an image while the vehicle is running. An identification target area extracting unit for extracting an identification target area from the inside, and whether or not an object exists in the identification target area from the image feature amount in the identification target area extracted by the identification target area extracting unit. Object identification means for identifying each type of the object, and the object identification means performs machine learning in which a feature data group as the image feature amount is input and presence / absence information of the object is output. And the classifier is selected according to the type of the target object from among a plurality of sub-regions constituting each learning sample image used for the machine learning. From one image of the sub-region, and wherein the inputting create the feature data group.

  Thus, the discriminator as the object discriminating means is an image of at least one sub-area selected according to the type of the object out of a plurality of sub-areas constituting each learning sample image used for machine learning. Since the feature data group is created and input, image information of sub-regions suitable for the shape of the projected image of the object can be selectively employed for the learning process, The learning accuracy can be improved regardless of the type. Then, by excluding the remaining sub-regions from the learning process, it is possible to prevent over-learning for image information other than the projected image of the object that acts as a disturbance factor during the identification process, and the identification accuracy of the object Can be improved. In addition, since the scene of the captured image obtained from the imaging means mounted on the vehicle changes every moment, it is particularly effective.

  Moreover, it is preferable that the said object identification means identifies for each moving direction of the said object whether the said object exists in the said identification object area | region. Based on the tendency that the image shape on the captured image changes according to the moving direction, the learning / identification accuracy of the object is further improved by identifying each moving direction.

  Furthermore, it is preferable that the identification target area extracting unit extracts the identification target area having a size proportional to a distance from the imaging unit to the target. As a result, the relative magnitude relationship between the target object and the identification target region becomes constant regardless of the distance, and the influence of disturbance factors (image information other than the projected image of the target object) is uniformly suppressed to uniformly Learning / identification accuracy is further improved.

  Furthermore, it is preferable that the image feature amount includes a luminance gradient direction histogram in space. Because the fluctuation in the brightness gradient direction due to the exposure amount of imaging is small, it is possible to accurately grasp the characteristics of the target object, and stable identification accuracy even in outdoor environments where the intensity of ambient light changes from moment to moment Is obtained.

  Furthermore, it is preferable that the image feature amount includes a luminance gradient direction histogram in time and space. By considering the luminance gradient direction in time as well as in space, it becomes easy to detect and track an object over a plurality of captured images acquired in time series.

  According to the vehicle periphery monitoring apparatus according to the present invention, the discriminator as the object discriminating unit is selected according to the type of the object out of the plurality of sub-regions constituting each learning sample image provided for machine learning. Since the feature data group is created and input from the at least one sub-region image, image information of the sub-region suitable for the shape of the projected image of the object is selectively selected for the learning process. The learning accuracy can be improved regardless of the type of the object. Then, by excluding the remaining sub-regions from the learning process, it is possible to prevent over-learning for image information other than the projected image of the object that acts as a disturbance factor during the identification process, and the identification accuracy of the object Can be improved. In addition, since the scene of the captured image obtained from the imaging means mounted on the vehicle changes every moment, it is particularly effective.

It is a block diagram which shows the structure of the vehicle periphery monitoring apparatus which concerns on this embodiment. It is a schematic perspective view of the vehicle carrying the vehicle periphery monitoring apparatus shown in FIG. 3A and 3B are image diagrams showing an example of a captured image acquired by imaging using a camera. It is a flowchart with which it uses for description of the learning process by the discriminator shown in FIG. It is an image figure which shows an example of the learning sample image containing a crossing pedestrian. It is a schematic explanatory drawing regarding the definition method of each sub area | region. It is a detailed block diagram of the discriminator shown in FIG. FIG. 8A is a schematic diagram showing a typical contour image obtained from a large number of learning sample images including crossing pedestrians. FIG. 8B is a schematic explanatory diagram illustrating a non-mask area and a mask area common to each learning sample image including a crossing pedestrian. FIG. 9A is a schematic diagram showing a typical contour image obtained from a large number of learning sample images including facing pedestrians. FIG. 9B is a schematic explanatory diagram showing a non-mask area and a mask area common to each learning sample image including a face-to-face pedestrian. 2 is a flowchart provided for explaining the operation of the ECU shown in FIG. It is a schematic explanatory drawing showing the positional relationship of a vehicle, a camera, and a human body. It is a schematic explanatory drawing regarding the determination method of an identification object area | region. It is a detailed block diagram of the target object identification part shown in FIG. 14A to 14C are schematic explanatory diagrams regarding a method for calculating the HOG feature amount. FIG. 15A is a schematic explanatory diagram relating to a method for calculating the HOG feature amount. FIG. 15B is a schematic explanatory diagram regarding a method of calculating an STHOG feature amount.

  Hereinafter, preferred embodiments of a vehicle periphery monitoring device according to the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a block diagram illustrating a configuration of a vehicle periphery monitoring device 10 according to the present embodiment. FIG. 2 is a schematic perspective view of the vehicle 12 on which the vehicle periphery monitoring device 10 shown in FIG. 1 is mounted.

  As shown in FIGS. 1 and 2, the vehicle periphery monitoring apparatus 10 includes a color camera (hereinafter simply referred to as “camera 14”) that captures a color image (hereinafter referred to as a captured image Im) including a plurality of color channels, and A vehicle speed sensor 16 for detecting the vehicle speed Vs of the vehicle 12, a yaw rate sensor 18 for detecting the yaw rate Yr of the vehicle 12, a brake sensor 20 for detecting the brake pedal operation amount Br by the driver, and the vehicle periphery monitoring device 10 An electronic control device (hereinafter referred to as “ECU 22”) to be controlled, a speaker 24 for issuing an alarm or the like by sound, and a display device 26 for displaying a captured image output from the camera 14 are provided.

  The camera 14 is a camera that mainly uses light having a wavelength in the visible light region, and functions as an imaging unit that images the periphery of the vehicle 12. The camera 14 has a characteristic that the output signal level increases as the amount of light reflected on the surface of the subject increases, and the luminance (for example, RGB value) of the image increases. As shown in FIG. 2, the camera 14 is fixedly disposed (mounted) at a substantially central portion of the front bumper portion of the vehicle 12.

  The imaging means for imaging the periphery of the vehicle 12 is not limited to the above configuration example (so-called monocular camera), and may be, for example, a compound eye camera (stereo camera). Further, an infrared camera may be used instead of the color camera, or both may be provided. Further, in the case of a monocular camera, another ranging means (radar apparatus) may be provided.

  Returning to FIG. 1, the speaker 24 outputs an alarm sound or the like in response to a command from the ECU 22. The speaker 24 is provided on a dashboard (not shown) of the vehicle 12. Alternatively, instead of the speaker 24, an audio output function provided in another device (for example, an audio device or a navigation device) may be used.

  The display device 26 (see FIGS. 1 and 2) is a HUD (head-up display) disposed on the front windshield of the vehicle 12 at a position that does not obstruct the driver's front view. The display device 26 is not limited to the HUD, but a display that displays a map or the like of a navigation system mounted on the vehicle 12 or a display (MID; multi-information display) that displays fuel consumption or the like provided in a meter unit or the like is used. can do.

  The ECU 22 basically includes an input / output unit 28, a calculation unit 30, a display control unit 32, and a storage unit 34.

  Signals from the camera 14, the vehicle speed sensor 16, the yaw rate sensor 18, and the brake sensor 20 are input to the ECU 22 side via the input / output unit 28. Each signal from the ECU 22 is output to the speaker 24 and the display device 26 via the input / output unit 28. The input / output unit 28 includes an A / D conversion circuit (not shown) that converts an input analog signal into a digital signal.

  The calculation unit 30 performs calculations based on the signals from the camera 14, the vehicle speed sensor 16, the yaw rate sensor 18, and the brake sensor 20, and generates signals for the speaker 24 and the display device 26 based on the calculation results. The calculation unit 30 functions as a distance estimation unit 40, an identification target region determination unit 42 (identification target region extraction unit), an object identification unit 44 (subject identification unit), and an object detection unit 46. Here, the target object identification unit 44 is configured by a classifier 50 generated using machine learning that receives a feature data group as an image feature amount and outputs the presence / absence information of the target object.

  The function of each unit in the calculation unit 30 is realized by reading and executing a program stored in the storage unit 34. Alternatively, the program may be supplied from the outside via a wireless communication device (mobile phone, smartphone, etc.) not shown.

  The display control unit 32 is a control circuit that drives and controls the display device 26. The display control unit 32 drives the display device 26 by outputting a signal used for display control to the display device 26 via the input / output unit 28. Thereby, the display apparatus 26 can display various images (captured image Im, a mark, etc.).

  The storage unit 34 includes an imaging signal converted into a digital signal, a RAM (Random Access Memory) that stores temporary data used for various arithmetic processes, and a ROM (Read Only Memory) that stores an execution program, a table, a map, or the like. ) Etc.

  The vehicle periphery monitoring apparatus 10 according to the present embodiment is basically configured as described above. An outline of the operation of the vehicle periphery monitoring device 10 will be described below.

  The ECU 22 converts the analog video signal output from the camera 14 into a digital signal at predetermined frame clock intervals / cycles (for example, 30 frames per second) and temporarily captures it in the storage unit 34.

  As shown in FIG. 3A, it is assumed that a captured image Im of the first frame is obtained at an imaging time point T = T1. In the illustrated example, in the captured image Im, a road area (hereinafter simply referred to as “road 60”) on which the vehicle 12 travels, and a plurality of utility pole areas (hereinafter simply referred to as “electric poles”) installed along the road 60 at substantially equal intervals. 62 ”) and a crossing pedestrian area on the road 60 (hereinafter simply referred to as“ crossing pedestrian 64 ”).

  As shown in FIG. 3B, it is assumed that a captured image Im of the second frame is obtained at an imaging time point T = T2 (> T1). In the captured image Im of this figure, as in the captured image Im shown in FIG. 3A, a road 60, a plurality of utility poles 62, and a crossing pedestrian 64 exist. However, as the frame interval time elapses, the relative positional relationship between the camera 14 and each object changes every moment. Thereby, even if the first and second frames (captured image Im) are in the same field angle range, each object is imaged in a different form (shape, size, or color).

  Then, the ECU 22 performs various arithmetic processes on the captured image Im (the front image of the vehicle 12) read from the storage unit 34. The ECU 22 (particularly the arithmetic unit 30) comprehensively considers the processing results for the captured image Im, and signals (vehicle speed Vs, yaw rate Yr, and operation amount Br) that indicate the traveling state of the vehicle 12 as necessary. A human body, an animal, or the like existing in front is detected as an object to be monitored (hereinafter referred to as “monitoring object” or simply “object”).

  When the calculation unit 30 determines that there is a high possibility that the vehicle 12 is in contact with the monitoring target, the ECU 22 controls each output unit of the vehicle periphery monitoring device 10 in order to call the driver's attention. For example, the ECU 22 outputs an alarm sound (for example, a beeping sound) via the speaker 24 and emphasizes the part of the monitored object in the captured image Im visualized on the display device 26. Display.

  Next, the learning process by the discriminator 50 shown in FIG. 1 will be described in detail with reference to the flowchart of FIG. As a machine learning method, any algorithm of supervised learning, unsupervised learning, and reinforcement learning may be employed. In the present embodiment, “supervised learning” in which learning is performed based on a data set given in advance will be described with a specific example.

  In step S11, learning data used for machine learning is collected. Here, the learning data is a data set of a learning sample image including (or not including) the target object and the type of the target object (including the attribute “no target object”). Examples of types of objects include human bodies, various animals (specifically, mammals such as deer, horses, sheep, dogs and cats, birds, etc.), artificial structures (specifically, vehicles, signs, utility poles) , Guardrails, walls, etc.).

  FIG. 5 is an image diagram showing an example of learning sample images 74 and 76 including crossing pedestrians 70 and 72. At the approximate center of the learning sample image 74, a projected image of a human body that crosswalks from the left side to the right side (hereinafter referred to as a crossing pedestrian 70) is displayed. In addition, a projected image of a human body that crosswalks from the right front side to the left back side (hereinafter referred to as crossing pedestrian 72) is displayed at the approximate center of another learning sample image 76. Each of the learning sample images 74 and 76 in this example corresponds to a correct image including the object. Each learning sample image collected may include an incorrect image that does not include an object. Further, in order to unify the data format input to the discriminator 50 (see FIG. 1) side, it is preferable that the learning sample images 74 and 76 have an image region 80 whose shapes match or are similar to each other.

  In step S <b> 12, a plurality of sub-regions 82 are defined by dividing the image region 80 of each learning sample image 74, 76. In the example of FIG. 6, the rectangular image region 80 is equally divided into a lattice shape with eight rows and six columns regardless of the size. That is, 48 sub-regions 82 having the same shape are defined in the image region 80, respectively.

  In step S13, a learning architecture of the classifier 50 is constructed. Examples of the learning architecture include a boosting method, SVM (Support Vector machine), neural network, EM (Expectation Maximization) algorithm, and the like. In this embodiment, AdaBoost, which is a kind of boosting method, is applied.

  As shown in FIG. 7, the discriminator 50 includes N (N is a natural number of 2 or more) feature data generators 90, N weak learners 92, a weight updater 93, and a weight calculator 94. And a sample load updater 95.

  In this figure, for convenience of explanation, the first data generator, the second data generator, the third data generator,..., The Nth data generator are sequentially arranged from the top with respect to each feature data generator 90. It is written. Similarly, the first weak learner, the second weak learner, the third weak learner,..., And the Nth weak learner are written in order from the top for each weak learner 92.

  As shown in the figure, one feature data generator 90 (for example, the first data generator) and one weak learner 92 (for example, the first weak learner) are alternatively connected, N subsystems are constructed. The output side (each weak learner 92) of N subsystems is connected to the input side of the weight updater 93, and the weighting calculator 94 and the sample load updater 95 are connected in series to the output side. It is connected. The detailed operation of the classifier 50 during machine learning will be described later.

  In step S14, the mask conditions for the sub-region 82 are determined for each type of object. Here, the mask condition is a selection as to whether or not to adopt an image in each sub-region 82 when N feature data (hereinafter referred to as a feature data group) is generated from one learning sample image 74. Means a condition.

  For example, when the captured image Im actually obtained by mounting the camera 14 on the vehicle 12 is used as the learning sample images 74 and 76 (correct images), various structures may be reflected behind the object. In the learning sample image 74 shown in FIG. 5, the background portion 78 excluding the crossing pedestrian 70 shows another three human bodies, a road surface, a building wall, and the like.

  During machine learning, if a feature data group including not only the crossing pedestrian 70 but also the background portion 78 is created, overlearning related to the background portion 78 may occur due to the tendency of the database that stores the learning sample images 74 and the like. In this case, the image information of the background part 78 acts as a disturbance factor (noise information) that reduces the learning / identification accuracy of the crossing pedestrian 70 as the object. Therefore, it is effective to select only the sub-region 82 suitable for the identification process from among all 48 sub-regions 82 and to learn using the created feature data group.

  FIG. 8A is a schematic diagram showing a typical contour image 100 obtained from a large number of learning sample images 74 and 76 including crossing pedestrians 70 and 72. Specifically, a contour extraction image (not shown) in which the contour of each object is extracted by performing known edge extraction processing on each learning sample image 74 or the like using an image processing device (not shown). Create each one. Here, each contour extraction image represents a part from which the contour of the object has been extracted in white, and represents a part from which the outline has not been extracted in black.

  Then, by using various image processing (for example, averaging processing) for each contour extraction image using an image processing device (not shown), a contour image typical for each learning sample image 74 (typical contour image 100). ) Here, similar to each contour extraction image, the typical contour image 100 represents a part from which the contour of the object has been extracted in white, and represents a part from which the contour has not been extracted in black. That is, the typical contour image 100 corresponds to an image representing the contour of an object (crossing pedestrians 70 and 72) included in common in a large number of learning sample images 74 and the like.

  For example, the sub region 82 whose contour feature amount exceeds a predetermined threshold is adopted as a calculation target, and the sub region 82 whose contour feature amount falls below a predetermined threshold is excluded from the calculation target. As a result, as shown in FIG. 8B, among the 48 sub-regions 82 defined in the image region 80, a set of 20 sub-regions 82 (region showing a partial image of the typical contour image 100) is obtained. It is assumed that the non-mask area 102 is determined. Of the 48 sub-regions 82, the remaining 28 sub-regions 82 (regions filled in with white) are determined as the mask region 104.

  FIG. 9A is a schematic diagram showing a typical contour image 106 obtained from a large number of learning sample images including a face-to-face pedestrian. The method for creating the typical contour image 106 is the same as that for the typical contour image 100 in FIG.

  FIG. 9B is a schematic explanatory diagram showing a non-mask area 108 and a mask area 110 that are common to each learning sample image including a face-to-face pedestrian. Since the determination method of the mask area 110 is the same as that of the mask area 104 in FIG. 8B, the description thereof is omitted.

  As can be understood from FIGS. 8B and 9B, the mask areas 104 and 110 are different even though the object is the same (pedestrian). More specifically, the mask areas 104 and 110 differ depending on whether or not the four sub-areas 82 (see FIG. 9B) with hatching are mask targets. This is because the degree of change in the image shape due to walking motion (movement of shaking hands and feet) varies depending on the moving direction of the pedestrian. In this way, learning may be performed for each moving direction of the object based on the tendency of the image shape on the image to change according to the moving direction. The moving direction may be any of a transverse direction (more specifically, right direction and left direction), a facing direction (more specifically, near side direction and back direction), and an oblique direction with respect to the image plane.

  In step S15, machine learning is performed by sequentially inputting a large number of learning data collected in step S11 to the discriminator 50. Hereinafter, it will be described in detail with reference to FIG.

  First, the discriminator 50 inputs the learning sample image 74 including the crossing pedestrian 70 among the collected learning data to the feature data generator 90 side. Then, each feature data generator 90 creates each feature data (collectively, feature data group) by performing specific arithmetic processing on the learning sample image 74 according to the mask condition determined in step S14. As a specific method of the mask processing, the calculation may be performed without using the values of all pixels belonging to the mask area 104 (see FIG. 8B), or the values of all the pixels described above are replaced with predetermined values (for example, 0). The feature data may be substantially invalidated by calculating later.

  Each weak learner 92 (i-th weak learner; 1 ≦ i ≦ N) is predetermined for each feature data (i-th feature data) acquired from the feature data generator 90 (i-th data generator). Each output result (i-th output result) is obtained by performing the above calculation. The weight updater 93 receives the first to Nth output results acquired from the weak learners 92, and the object information 96 that is the presence / absence information of the object in the collected learning data. Enter. Here, the object information 96 indicates that the crossing pedestrian 70 is included in the learning sample image 74.

  Thereafter, the weight updater 93 selects one weak learner 92 that has obtained an output result that minimizes the amount of error from the output value corresponding to the object information 96, and updates the weighting coefficient α so as to increase. The quantity Δα is determined. The weighting calculator 94 updates the weighting coefficient α by adding the update amount Δα supplied from the weight updater 93. At the same time, the sample load updater 95 updates a load (hereinafter referred to as a sample load 97) previously applied to the learning sample image 74 and the like based on the updated weighting coefficient α and the like.

  In this manner, the learning data is input, the weighting coefficient α is updated, and the sample load 97 is sequentially updated, and the discriminator 50 performs machine learning until the convergence condition is satisfied (step S15).

  As described above, by mounting the machine-learned discriminator 50 in the vehicle periphery monitoring device 10 (see FIG. 1), whether at least one type of object exists in the discrimination target area of the captured image Im. An object identification unit 44 that can identify whether or not is constructed.

  Next, the detailed operation of the vehicle periphery monitoring device 10 will be described with reference to the flowchart of FIG. Note that this processing flow is executed for each imaging frame when the vehicle 12 is traveling.

  In step S21, the ECU 22 acquires a captured image Im that is an output signal in front of the vehicle 12 (predetermined angle of view range) captured by the camera 14 for each frame. For example, as shown in FIG. 3A, it is assumed that a captured image Im in units of frames is obtained at an imaging time T = T1. Then, the ECU 22 temporarily stores the acquired captured image Im in the storage unit 34. For example, when an RGB camera is used as the camera 14, the obtained captured image Im is a multi-gradation image composed of three color channels.

  In step S22, the distance estimation unit 40 estimates the distance Dis from the vehicle 12 by calculating the elevation angle of the vehicle 12 using the captured image Im acquired in step S21.

  FIG. 11 is a schematic explanatory diagram showing the positional relationship between the vehicle 12, the camera 14, and the human body M. Here, it is assumed that the vehicle 12 on which the camera 14 is mounted and the human body M as an object are present on a flat road surface S. A contact point between the human body M and the road surface S is Pc, an optical axis of the camera 14 is L1, and a straight line connecting the optical center C of the camera 14 and the contact point Pc is L2.

For example, the angle (elevation angle) formed by the optical axis L1 of the camera 14 with respect to the road surface S is β, the angle formed by the straight line L2 with respect to the straight line L1 is γ, and the height of the camera 14 with respect to the road surface S is Hc. Suppose that In this case, the distance Dis is calculated by the following equation (1) using the angles β, γ, and the height Hc from a geometrical consideration.
Dis = Hc / tan (β + γ) (1)

  In this way, the distance estimation unit 40 can estimate the distance Dis corresponding to each position of the road surface S (the road 60 in FIG. 3A) on the captured image Im. Alternatively, the distance estimation unit 40 estimates the distance Dis using a known method such as SfM (Structure from Motion) in consideration of the posture change between the road surface S and the camera 14 accompanying the motion of the vehicle 12. Also good. In addition, when the vehicle periphery monitoring apparatus 10 is provided with a distance measuring sensor, you may measure the distance Dis using this.

  In step S23, the identification target area determination unit 42 determines the size and the like of the identification target area 122 that is an image area to be identified. In the present embodiment, the identification target area determination unit 42 determines the size or the like of the identification target area 122 according to the distance Dis estimated in step S22 and / or the vehicle speed Vs acquired from the vehicle speed sensor 16. A specific example will be described with reference to FIG.

  As illustrated in FIG. 12, a position on the captured image Im corresponding to a contact point Pc (see FIG. 11) between the road surface S (road 60) and the human body M (crossing pedestrian 64) is set as a reference position 120. And the rectangular identification object area | region 122 is set so that all the crossing pedestrians 64 may be included.

  The identification target area determination unit 42 determines the size of the identification target area 122 so as to be proportional to the distance Dis from the camera 14 (optical center C in FIG. 11) to the target. When it is assumed that the crossing pedestrian 64f illustrated by a broken line exists at the reference position 124 on the road surface S (the road 60), an identification target area 126 similar to the identification target area 122 is set. As a result, the relative magnitude relationship between the crossing pedestrian 64 (64f) and the identification target region 122 (126) becomes constant regardless of the distance Dis, and the disturbance factor (image information other than the projected image of the target object) The learning and identification accuracy of the object is further improved by suppressing the influence uniformly.

  Then, the identification target area determination unit 42 also determines a designated area 128 that is a target range of a raster scan described later. For example, Dis1 that is a distance in which the object is reliably detected within the normal operation range is determined as the lower limit value, and Dis2 that is a distance that does not cause a collision with the object immediately within the normal operation range is set as the upper limit value. May be determined as As described above, by omitting scanning in a part of the captured image Im, not only can the calculation amount and calculation time of the identification process be reduced, but also erroneous detection itself that may occur in a region other than the designated region 128. Can be eliminated.

  Note that the identification target area determination unit 42 may appropriately change the shape of the identification target area 122 according to the type of the target object. In this case, as in the case of the present embodiment, the size may be determined in proportion to the distance Dis.

  Further, the identification target area determination unit 42 may change the size of the identification target area 122 according to the height of the target object even at the same distance Dis. Thereby, it is possible to set an appropriate size according to the height of the object, and the learning / identification accuracy of the object is further improved by uniformly suppressing the influence of the disturbance factor.

  In step S24, the arithmetic unit 30 starts a raster scan of the captured image Im within the designated area 128 determined in step S23. Here, the raster scan refers to a method of successively identifying the presence or absence of an object while moving the reference position 120 (pixels in the captured image Im) in a predetermined direction. Hereinafter, the identification target area determination unit 42 sequentially determines the reference position 120 currently being scanned and the position / size of the identification target area 122 identified from the reference position 120.

  In step S <b> 25, the object identification unit 44 identifies whether or not at least one type of object exists in the determined identification object region 122.

  As illustrated in FIG. 13, the object identification unit 44 is a classifier 50 (see FIG. 7) generated using machine learning. In the weighting calculator 94, an appropriate weighting coefficient αf obtained by machine learning (step S15 in FIG. 4) is preset.

  The object identifying unit 44 inputs an evaluation image 130 having an image area 80 including the identification target area 122 to each feature data generator 90 side. Here, in order to increase the identification accuracy by the object identification unit 44, necessary image processing such as normalization processing (gradation processing / enlargement / reduction processing), alignment processing, and the like may be appropriately performed on the image of the identification target region 122. Good. The object identification unit 44 includes each feature data generator 90, each weak learner 92, a weighting calculator 94, and an integrated learner that applies a step function to the weighted output result acquired from the weighting calculator 94. The evaluation image 130 is sequentially processed via 98, and an identification result indicating that the crossing pedestrian 64 exists in the identification target area 122 is output. In this case, the object discriminating unit 44 functions as a strong discriminator having high discrimination performance by combining N weak discriminators (weak learners 92).

  Each feature data generator 90 calculates an image feature amount (that is, the above-described feature data group) in each sub-region 82 using the same calculation method as in the learning process (see FIG. 7).

  By the way, various known methods can be used as the image feature amount calculation method. Hereinafter, the HOG (Histograms of Oriented Gradient) feature amount indicating the intensity and gradient characteristics of the luminance in the local region of the image will be described.

  As shown in FIG. 14A, it is assumed that one block which is a histogram creation unit is selected from the image area 80. In order to facilitate understanding of the technology, each block is defined below corresponding to each sub-region 82. For example, it is assumed that the sub-region 82 as a block is composed of a total of 36 pixels 84, 6 pixels vertically and 6 pixels horizontally.

As shown in FIG. 14B, a two-dimensional gradient (Ix, Iy) of luminance is calculated for each pixel 84 constituting the block. In this case, the gradient intensity I and the spatial luminance gradient angle θ are calculated according to the following equations (2) and (3).
I = (Ix ² + Iy ² ) ^1/2 (2)
θ = tan ⁻¹ (Iy / Ix) (3)

  The arrows written in each grid in the first row illustrate the direction of the planar luminance gradient. Actually, the gradient intensity I and the spatial luminance gradient angle θ are calculated for all the pixels 84, but the illustration of arrows in the second and subsequent rows is omitted.

  As shown in FIG. 14C, a histogram for the spatial luminance gradient angle θ is created for each block. The horizontal axis of the histogram is the spatial luminance gradient angle θ (eight divisions in this example), and the vertical axis of the histogram is the gradient intensity I. In this case, a histogram for each block is created based on the gradient intensity I shown in Equation (2).

  As illustrated in FIG. 15A, the HOG feature amount of the evaluation image 130 is obtained by connecting the histogram for each block (spatial luminance gradient angle θ in the example of FIG. 14C) in a predetermined order, for example, ascending order.

  Thus, the image feature amount may include a luminance gradient direction histogram (HOG feature amount) in space. Because the fluctuation of the luminance gradient direction (θ) due to the exposure amount of imaging is small, it is possible to accurately capture the characteristics of the target object, and it is stable even in outdoor environments where the intensity of ambient light changes from moment to moment Identification accuracy can be obtained.

Further, as the image feature amount, an STHOG (Spatio-Temporal Histograms of Oriented Gradient) feature amount in which the definition of the HOG feature amount is expanded may be used. In this case, a three-dimensional gradient (Ix, Iy, It) of luminance is calculated for each pixel 84 constituting the block using a plurality of captured images Im acquired in time series, as in FIG. 14B. Is done. Here, the gradient intensity I and the temporal luminance gradient angle φ are calculated according to the following equations (4) and (5).
I = (Ix ² + Iy ² + It ² ) ^1/2 (4)
φ = tan ⁻¹ {It / (Ix ² + Iy ² ) ^1/2 } (5)

  As shown in FIG. 15B, an STHOG feature quantity that is a brightness gradient histogram in space-time is obtained by further connecting a histogram of temporal brightness gradient angle φ to the HOG feature quantity. Thus, not only the luminance gradient direction (θ) in space but also the luminance gradient direction (φ) in time is taken into consideration, thereby detecting an object over a plurality of captured images Im acquired in time series.・ Easy tracking.

  In the case of the STHOG feature value, similarly to the HOG feature value, a histogram for each block is created based on the gradient intensity I shown in the equation (2).

  In this way, the object identification unit 44 identifies whether or not at least one type of object exists in the determined identification object region 122 (step S25). Thereby, not only the kind of the object including the human body M but also the moving direction including the transverse direction and the facing direction are identified.

  In step S <b> 26, the identification target area determination unit 42 determines whether all the scans in the designated area 128 have been completed. When it is determined that it is not completed (step S26: NO), the process proceeds to the next step (S27).

  In step S <b> 27, the identification target area determination unit 42 changes the position or size of the identification target area 122. Specifically, the identification target area determination unit 42 moves the reference position 120 that is the scan target by a predetermined amount (for example, one pixel) in a predetermined direction (for example, the right direction). When the distance Dis changes, the size of the identification target area 122 is also changed. Furthermore, considering that the typical value of the body length or the body width varies depending on the type of the object, the identification target region determination unit 42 may change the size of the identification target region 122 according to the type of the target.

  Thereafter, returning to step S25, the arithmetic unit 30 sequentially repeats steps S25 to S27 until all the scans in the designated area 128 are completed. When it is determined that the processing has been completed (step S26: YES), the calculation unit 30 ends the raster scan of the captured image Im (step S28).

  In step S29, the object detection unit 46 detects an object present in the captured image Im. The identification result for a single frame may be used, or the motion vector for the same object can be calculated by considering the identification results for a plurality of frames.

  In step S30, the ECU 22 causes the storage unit 34 to store data necessary for the next calculation process. For example, the distance Dis created in step S22, the attribute of the object (crossing pedestrian 64 in FIG. 3A, etc.) obtained in step S25, the reference position 120, and the like can be given.

  By sequentially executing this operation, the vehicle periphery monitoring device 10 can monitor an object (for example, the human body M in FIG. 11) existing in front of the vehicle 12 at a predetermined time interval.

  As described above, the vehicle periphery monitoring device 10 according to the present invention is mounted on the vehicle 12 and acquired by the camera 14 that acquires the captured image Im in the vicinity of the vehicle 12 by capturing an image while the vehicle 12 is traveling. The identification target area determination unit 42 that extracts the identification target areas 122 and 126 from the captured image Im, and the object (for example, in the identification target areas 122 and 126) based on the extracted image feature amounts in the identification target areas 122 and 126. And an object identifying unit 44 for identifying whether or not there is a crossing pedestrian 64) for each type of object. The target object identification unit 44 is a classifier 50 generated using machine learning that receives a feature data group as an image feature amount and outputs target presence / absence information.

  The discriminator 50 as the object discriminating means includes at least one sub selected according to the type of the object out of the plurality of sub areas 82 constituting the learning sample images 74 and 76 used for machine learning. Since the feature data group is created and input from the image of the region 82 (non-mask regions 102 and 108), the image information of the sub-region 82 suitable for the shape of the projected image of the object is obtained for the learning process. The learning accuracy can be improved regardless of the type of object. Further, by excluding the remaining sub-region 82 (mask regions 104 and 110) from the learning process, it is possible to prevent over-learning with respect to image information other than the projected image of the object that acts as a disturbance factor during the identification process. Yes, the identification accuracy of the object can be improved. Note that the scene (background, weather, road surface pattern, etc.) of the captured image Im obtained from the camera 14 mounted on the vehicle changes from moment to moment and is particularly effective.

  In addition, this invention is not limited to embodiment mentioned above, Of course, it can change freely in the range which does not deviate from the main point of this invention.

  In the present embodiment, the above-described identification process is performed on the captured image Im obtained by the monocular camera (camera 14), but it goes without saying that the same effect can also be obtained with a compound eye camera (stereo camera). .

  In the present embodiment, the learning process and the identification process by the object identification unit 44 (identifier 50) are performed separately, but both processes may be provided so as to be executed in parallel.

DESCRIPTION OF SYMBOLS 10 ... Vehicle periphery monitoring apparatus 12 ... Vehicle 14 ... Camera 22 ... ECU
DESCRIPTION OF SYMBOLS 30 ... Operation part 34 ... Memory | storage part 40 ... Distance estimation part 42 ... Identification object area | region determination part 44 ... Object identification part 46 ... Object detection part 50 ... Discriminator 64, 70, 72 ... Crossing pedestrian 74, 76 ... Learning Sample image 78 ... Background portion 80 ... Image area 82 ... Sub-area 90 ... Feature data generator 92 ... Weak learner 94 ... Weight calculator 96 ... Object information 98 ... Integrated learner 100, 106 ... Typical contour images 102, 108 ... Non-mask area 104, 110 ... Mask area 120, 124 ... Reference position 122, 126 ... Identification target area 128 ... Designated area 130 ... Evaluation image

Claims (5)

An imaging means mounted on a vehicle for acquiring a captured image in the vicinity of the vehicle by capturing an image while the vehicle is running;
An identification target area extracting means for extracting an identification target area from the captured image acquired by the imaging means;
Object identification means for identifying, for each type of the object, whether or not an object exists in the identification object area from the image feature amount in the identification object area extracted by the identification object area extraction means. Prepared,
The target object identification means is a classifier generated using machine learning, wherein a feature data group as the image feature amount is input and presence / absence information of the target object is output.
The classifier includes the feature data group from images of at least one sub-region selected according to the type of the target object among a plurality of sub-regions constituting each learning sample image used for the machine learning. A vehicle periphery monitoring device characterized by creating and inputting a vehicle. The vehicle periphery monitoring device according to claim 1,
The vehicle object monitoring device, wherein the object identifying means identifies whether the object exists in the identification object area for each moving direction of the object. The apparatus according to claim 1 or 2,
The vehicle periphery monitoring apparatus, wherein the identification target area extracting unit extracts the identification target area having a size proportional to a distance from the imaging unit to the target. The device according to any one of claims 1 to 3,
A vehicle periphery monitoring device, wherein the image feature amount includes a luminance gradient direction histogram in space. The device according to any one of claims 1 to 3,
A vehicle periphery monitoring apparatus, wherein the image feature amount includes a luminance gradient direction histogram in time and space.

Images