JP6156212B2 - Object detection device - Google Patents

Description

  The present invention relates to a technique for detecting an object from a distorted image.

  2. Description of the Related Art Conventionally, an object is detected using an object model from an image captured around a vehicle, and the detection result is used for control such as driving assistance. At that time, in order to widen the detection range, it is also considered to use a fisheye camera with a wide angle of view. However, since the image picked up by the fisheye camera is distorted, if the object detection technique based on an image without distortion obtained from a normal camera is applied as it is, the detection accuracy of the object is lowered.

  On the other hand, a conventional technique is known in which distortion correction processing is performed on a distorted image and object detection is performed using the corrected image. In this conventional technology, when correcting a distorted image, the calculation formula can be speeded up by decomposing the correction formula into a conversion formula for the overhead view, which makes the correction formula relatively simple, and a conversion formula for the overhead view. Has also been proposed (see Patent Document 1).

Japanese Patent No. 4243767

  By the way, in distortion correction processing, it is necessary to perform calculation for correcting distortion for each pixel of the corrected image. For example, an image with a resolution used in a normal in-vehicle camera has, for example, about 80,000 pixels in QVGA, and enormous calculation is required to perform correction processing on all the pixels. In particular, an in-vehicle microcomputer with poor calculation capability has a problem that it is difficult to apply such a method because the load becomes excessive.

Further, the distortion correction process described in Patent Document 1 has a problem in that recognition accuracy is lowered because appropriate correction is not performed in a peripheral portion where distortion is large.
The present invention has been made in view of these problems, and an object of the present invention is to provide a technique for reducing the processing load of object detection using a distorted image and improving the detection accuracy.

  The object detection apparatus of the present invention includes a target model storage unit, an image acquisition unit, a feature extraction unit, a model adjustment unit, and an object detection unit. The target model storage means is prepared for each object to be detected, each of which has a plurality of parts corresponding to a part of the object, and represents the relative position relationship between the parts. The target object model in which the part feature quantity that is the feature quantity representing the feature of the object is defined is stored. The image acquisition means acquires an image having distortion. The feature extraction unit extracts, based on the image acquired by the image acquisition unit, an image feature amount, which is a feature amount of the unit region, for each unit region cut out from the image. The model adjustment means uses a point obtained by projecting a search point set in the three-dimensional space to be imaged on the image plane as a projection search point, and an object model according to the degree of image distortion obtained for each projection search point. Adjust. The object detection means is associated with the object model by evaluating the similarity between the image feature amount extracted by the feature extraction means and the adjustment model that is the object model adjusted by the model adjustment means. Detects an object.

  As described above, according to the present invention, the object model is adjusted according to the image distortion at the projection search point to which the object model is applied instead of correcting the image distortion. The processing load can be reduced, and the detection accuracy of object detection using a distorted image can be improved.

  In addition, the code | symbol in the parenthesis described in the claim shows the correspondence with the specific means as described in embodiment mentioned later as one aspect, Comprising: The technical scope of this invention is limited is not.

  In addition to the object detection device described above, the present invention is realized in various forms such as various systems including the object detection device as components, a program for causing a computer to function as the object detection device, and an object detection method. be able to.

It is a block diagram which shows the whole structure of a driving assistance system. It is explanatory drawing which shows the structure of a target object model. It is explanatory drawing which shows the shape of a cell block and a cell. It is explanatory drawing which shows the rotational symmetry of the cell which comprises a cell block. It is explanatory drawing of a brightness | luminance gradient histogram, (a) is a figure which shows the correspondence of a bin and a gradient direction, (b) is a figure which shows an example of a brightness | luminance gradient histogram. It is a flowchart which shows the content of the identification target determination process which CPU of a target object identification part performs. It is explanatory drawing which shows the example of a setting of a search point. It is a flowchart which shows the detail of the model adjustment process performed in the identification target determination process. It is explanatory drawing which shows how to project on the image plane by a fisheye camera model. It is explanatory drawing which shows the calculation method of the part shift amount and part rotation amount which are used for adjustment of a target object model. It is explanatory drawing which shows the relationship between rotation of a cell block and the arrangement | sequence of the bin number of a brightness | luminance gradient histogram. It is the table | surface which illustrated the content of the corresponding | compatible table used for adjustment of part feature-value. It is explanatory drawing which shows the effect by adjusting a target object model. It is explanatory drawing which shows the problem in case a cell block is a rectangle.

Embodiments to which the present invention is applied will be described below with reference to the drawings.
[overall structure]
As shown in FIG. 1, a driving support system 1 to which the present invention is applied includes an imaging device 2 that images the front of the host vehicle and an image acquired by the imaging device 2 (hereinafter referred to as an input image). An image processing unit 3 that performs processing for detecting an object that is present ahead and is an identification target (hereinafter, referred to as a target), and controls various vehicle-mounted devices according to the processing result in the image processing unit 3 to provide predetermined driving assistance. And a driving support execution unit 4 to be executed.

  Among these, the imaging device 2 is composed of a fish-eye camera disposed on a vehicle body such as the vicinity of a front and rear bumpers and side mirrors. Therefore, the input image from the imaging device 2 to the image processing unit 3 is entirely distorted (see FIG. 13). In addition, the in-vehicle device to be controlled by the driving support execution unit 4 includes at least a monitor 41 that displays various images and a speaker 42 that outputs alarm sound and guidance sound.

  The image processing unit 3 is used for processing in the target object identifying unit 10 for identifying the target object from the input image and the target object identifying unit 10, and adjusting the target object model and target object model prepared for each target object. An identification information storage unit 20 that stores a correspondence table used for the communication, and a communication unit 30 that supplies the processing result of the object identification unit 10 to various on-vehicle devices connected by the in-vehicle LAN.

  The object identification unit 10 is formed of a known microcomputer including a CPU, a ROM, a RAM, and the like, and executes at least an object detection process described later. In addition to the object detection processing program, the ROM stores at least an object model prepared in advance for each object and a correspondence table used for adjustment of the object model.

[Object model]
As shown in FIG. 2 (a), the object model has a plurality of parts MPi corresponding to each part constituting the object (i = 1 to 6 in the figure) for each object (object) to be detected. It is defined using For example, when the object is a person, the head, the left arm, the right arm, the torso, the left leg, the right leg, and the like can be considered as the parts constituting the object.

  The object model is expressed by a feature amount (part feature amount) that expresses the feature of the part of the object corresponding to each part MPi and relative position information that defines a relative positional relationship between the parts MPi. Is done.

<Relative position information>
As shown in FIG. 2B, the relative position information includes the coordinates of the representative reference point Pc (i) set for each part MPi and the coordinates of a plurality of correction reference points Pa (i, j). Here, i is an identifier for identifying a part, and j is an identifier for identifying a plurality of correction reference points in the same part. As shown in FIG. 2A, these coordinates are set in advance in a model reference point MO (for example, a rectangular area MD set so as to include all the parts MPi constituting the object model). Consists of three-dimensional coordinates with the origin at the bottom center. However, in the three-dimensional coordinates, the horizontal direction is the X-axis, the height direction is the Y-axis, the depth direction is the Z-axis, the part MPi exists on the XY plane with Z = 0, and the model reference point MO is the origin of the coordinates It is set to become. That is, the relative positional relationship between the parts MPi constituting a certain object model is defined by a relative coordinate system independent for each object model.

  The representative reference point Pc (i) is a point representing the part MPi, and is set at the center of the part MPi, for example. The correction reference point Pa (i, j) is a point provided for calculating a part rotation amount (distortion correction parameter), which will be described later, with respect to the representative reference point Pc (i). It is set at the position translated along the axial direction. Here, two correction reference points Pa (i, 1) and Pa (i, 2) are located at positions parallel to the Y-axis direction and symmetrical positions with respect to the representative reference point Pc (i). Is set.

  For example, assuming the pedestrian's head as part MP1 of the object model with the object as a pedestrian, the coordinates of the representative reference point are set to Pc (1) = (1.7, 0, 0). The coordinates of the two reference points for correction are set to Pa (1,1) = (1.75,0,0) and Pa (1,2) = (1.65,0,0). The coordinates of the representative reference point exemplified here mean that the part MP1 representing the head is located at a height of 1.7 m, but considering that the height of the pedestrian is wide, the same kind of object A plurality of target models may be prepared for an object.

<Feature amount>
The feature amount is obtained by obtaining a feature of an edge image (luminance differential image) generated from an input image in a unit of a cell block (described later) composed of a plurality of cells, and is represented by a multidimensional vector. . Hereinafter, the generation method will be described.

  First, a luminance gradient is obtained for each pixel constituting the edge image. Specifically, the luminance gradients (dx, dy) in the X-axis direction and the Y-axis direction are obtained from the luminance values of the pixels adjacent to the pixel of interest vertically and horizontally, and the luminance gradient of the pixel of interest is calculated from the calculation result. The intensity v and the direction θ are obtained by equations (1) and (2).

  Next, an image is cut out in cell block units, and a luminance gradient histogram is generated for each of the cells constituting the cell block. As shown in FIG. 3, the cell block is set in a circular shape, and a total of 16 (= M) partial areas obtained by dividing the cell block into eight equal parts in the circumferential direction and two equal parts in the radial direction are cells. . The numbers in the figure are cell numbers that identify the cells. That is, as shown in FIG. 4, each cell constituting the cell block is arranged to be rotationally symmetric with respect to rotation at a predetermined angle (here, an integral multiple of 45 °). Here, a cell block having 16 cells is shown, but other cell arrangement methods may be used as long as they are rotationally symmetric.

  As shown in FIG. 5A, the brightness gradient histogram is associated with 8 predetermined angle ranges (45 °, which is the same as the predetermined angle that is rotationally symmetric with the cell block), each representing a different gradient direction. Bins (bin numbers 1 to 8). Then, as shown in FIG. 5B, a luminance gradient histogram for each cell is created by voting to which bin the luminance gradient obtained for each pixel belonging to the cell of interest belongs. Then, the feature amount is obtained by assigning each value of the N bins representing the brightness gradient histogram obtained for each of the M cells constituting the same cell block in a predetermined order (for example, the order of the cell numbers shown in FIG. 3). It is represented by a concatenated M × N-dimensional vector.

  In the following, the feature quantity generated from the input image is called an image feature quantity, and the feature quantity set for each part constituting the object model is called a part feature quantity. The part feature amount is set by learning using a large number of images taken for the part of the object associated with the part of interest, and using the image feature amount extracted from these images as teacher data. Since this type of learning is a well-known technique, a detailed description thereof is omitted.

[Identification target judgment processing]
The contents of the identification object determination process executed by the CPU of the object identification unit 10 will be described with reference to the flowchart shown in FIG. This process is started every time captured images are generated at a predetermined interval in the imaging apparatus 2.

  When this process is started, the CPU first acquires a captured image generated by the imaging device 2 as an input image (S110), and generates an edge image of the input image (S120). Next, the CPU obtains a luminance gradient of each pixel constituting the edge image, and calculates an image feature amount for each unit region (cell group) cut out from the edge image (S130).

Next, the CPU selects one object model (S140), and further selects one search point (S150).
As shown in FIG. 7, the search points are regularly set on the XZ plane (ground) of the three-dimensional space to be imaged by the imaging device 2. Here, a point (with a predetermined angle direction (0 °, ± α, ± 2α, ± 3α) and a predetermined distance (R, 2R, 3R, 4R) with respect to the front direction of the vehicle on which the driving support system 1 is mounted) A point indicated by a black dot in the drawing) is set as a search point Ps (k). Here, k is an identifier for identifying a search point. Note that the arrangement of the search points is not limited to this. For example, the search points may be set so that the projection points on the image plane are arranged in a lattice pattern at regular intervals.

  Returning to FIG. 6, the CPU executes model adjustment processing for adjusting the object model selected in the previous S140 in accordance with the search point Ps (k) selected in the previous S150 (S160). Is detected by collating the image feature quantity within a predetermined range centered on a point projected on the image plane (hereinafter referred to as a projection search point) with the object model (adjusted model) adjusted in S160. It performs (S170). In addition, since the technique which detects an object by collation with an image feature-value and a target object model is known, description is abbreviate | omitted for the detail.

  Next, the CPU determines whether or not the above-described processing of S150 to S170 has been executed for all search points Ps (k) (S180), and if there is an unprocessed search point Ps (k). Return to S150. On the other hand, if the processing has been executed for all the search points, it is further determined whether or not the above-described processing of S140 to S180 has been executed for all target models (S190), and unprocessed targets are determined. If there is an object model, the process returns to S140. On the other hand, if the processing has been executed for all the object models, the detection result of each object corresponding to each object model is output to the driving support execution unit 4 via the communication unit 30 (S200). End the process.

[Model adjustment processing]
Details of the model adjustment process executed in S160 will be described with reference to the flowchart shown in FIG.

  In this process, the CPU first selects one of a plurality of parts constituting the object model for the object model selected in the previous S140 (S310), and is selected in the previous S150. The representative reference point Pcs on the search point and the correction reference point Pas on the search point representing the three-dimensional coordinates of the representative reference point Pc and the correction reference point Pa when it is assumed that the object model exists on the search point (3 ) Calculate using equation (4) (S320). However, Ps (k) represents the three-dimensional coordinates of the selected search point.

Then, CPU is the search point on the representative reference point Pcs or on the searched point correction reference point Pas determined at S320 and the target point P, a point on the image plane the object point P is projected as a projection point p _I Then, the coordinates (x _p , y _p ) of the projection point p _I are calculated using the equations (5) and (6) (S330).

However, as shown in FIG. 9B, r is the distance from the image center O _I to the projection point p _I (image height considering image distortion), and ψ is the projection point p with respect to the horizontal line (X axis) on the image plane. Represents the angle of _I.

Equations (5) and (6) are derived from the fisheye camera model shown in FIG. Fisheye camera model, object point P, the camera center Oc, image center O _I, object point P, the projection point of the image distortion on the image plane of the object point P in the case of not considering P _I, taking into account the image distortions the projection point onto the image plane of the object point P in the case of those showing the relationship p _I. Here, the three-dimensional coordinates of the target point P (X, Y, Z), the angle of the projected point P _I and the optical axis phi, the focal length (the distance from the camera center Oc to the image center O _I) f, The pixel size is w (not shown).

As is clear from FIG. 9, the angle ψ is calculated by the equation (7). On the other hand, the image height r differs depending on the fish-eye camera system. For example, in the equidistant projection system, the image height r changes according to the angle φ as shown in the equation (8). However, the angle φ is calculated by the equation (9). That is, in the absence of image distortion camera, object point P is the line segment connecting the target point P and the camera center Oc is a point which intersects the image plane (virtual projection point) onto a P _I. On the other hand, in the camera with an image distortion such as a fisheye camera, object point P is located (actual projection point) offset from the virtual projection point P _I is projected in p _I. This shift is reflected in the image height r.

Returning to FIG. 8, next, the CPU adjusts the object model based on the coordinates of the projection point on the image plane of the representative reference point Pcs on the search point and the reference point Pas for correction on the search point calculated in S330. As an adjustment amount to be used, a part shift amount (Δx _c , Δy _c ) and a part rotation amount η are calculated (S340).

Specifically, as shown in FIG. 10A, the part shift amounts (Δx _c , Δy _c ) are obtained by setting the coordinates of the virtual projection point of the representative reference point Pcs on the search point to (X _c , Y _c ), The coordinate of the projection point is set to (x _c , y _c ), and the difference between the two is calculated by equation (10). As shown in FIG. 10B, the part rotation amount η is the actual projection of the representative reference point Pcs on the search point with the coordinates of the actual projection point of the correction point Pas on the search point being (x _a , y _a ). The angle formed by the line segment connecting the point and the actual projection point of the reference point Pas for correction on the search point with respect to the Y axis is calculated by the equation (11). As the part rotation amount η, when a plurality of correction reference points Pa are prepared, an average of angles calculated for each may be used. When the correction reference point Pa is set at a position shifted in the X-axis direction with respect to the representative reference point Pc, the angle formed by the line segment with respect to the X-axis may be obtained.

  Returning to FIG. 8, next, the CPU adjusts the part feature quantity by changing the arrangement of the elements of the multidimensional vector representing the part feature quantity in accordance with the part rotation amount η calculated in S330 (S350). .

  Specifically, the part feature amount of the object model is obtained by arranging the luminance gradient histogram of each cell constituting the cell block in the order of the cell number, and each element constituting the luminance gradient histogram of each cell is: As shown to Fig.11 (a), it arranges in order of the bin numbers 1-8. Here, assuming that the upward direction in the image to be compared with the cell block (hereinafter referred to as “comparison image”) is the reference direction, the gradient direction corresponding to bin number 1 matches the reference direction. Here, considering the case where the comparison image is tilted 45 ° to the left, in order to detect this correctly, as shown in FIG. 11B, it is necessary to apply the cell block by rotating it 45 ° to the left. is there. However, if the cell block is rotated, the correspondence between the bin number and the gradient direction in the luminance gradient histogram of each cell is lost. However, since each cell constituting the cell block is arranged so as to be rotationally symmetric with respect to rotation of an integral multiple of 45 °, the cell block is not rotated (see FIG. 11A). By using the generated luminance gradient histogram and shifting the order of bins of the luminance gradient histogram in accordance with the part rotation amount η, a part feature amount suitable for an inclined comparative image is obtained. Here, the bin numbers 1 to 8 are rearranged in the order of bin numbers 2 to 8 and 1.

  As an actual process, a correspondence table showing the correspondence between the part rotation amount η and the bin number array as shown in FIG. 12 is stored in the ROM, and the multidimensional data is displayed according to the correspondence shown in the correspondence table. What is necessary is just to change the arrangement order of each element which comprises a vector.

  Returning to FIG. 8, next, the CPU determines whether or not the above-described processing (S310 to S350) has been performed for all the parts constituting the object model selected in the previous S140 (S360). ) If there is a part that has not been subjected to the above-described processing, the process returns to S310, and if the above-described processing has been performed for all the parts, this processing ends.

[effect]
As described above, in the present embodiment, because the distortion correction of the input image is not performed on the distorted input image obtained from the fisheye camera, the object model is adjusted according to the distortion. Compared to a conventional apparatus that performs distortion correction processing on an input image, the processing amount required for object detection processing can be reduced.

  In the present embodiment, the distortion of the input image is extracted as the part shift amount and the part rotation amount, and the relative positional relationship of the parts constituting the object model is changed according to the part shift amount. The feature amount of each part is changed according to the amount of rotation.

  As a result, as shown in FIG. 13A, the positional relationship and part feature quantity of each part constituting the object model can be made accurate according to the degree of distortion at the application location. As shown in (b), the detection accuracy can be improved as compared with the case where the object model is applied without adjustment.

  In this embodiment, each cell constituting the cell block has a rotationally symmetric shape with respect to a predetermined rotation determined in relation to the setting of the bin of the luminance gradient histogram. A luminance gradient histogram corresponding to the part rotation amount η can be realized simply by changing the order of bins. As a result, the adjustment of the part feature amount according to the part rotation amount η can be realized without recalculating the part feature amount, and the processing load can be reduced in this respect as well.

  That is, for example, as shown in FIG. 14, when the cell block is configured by square cells arranged in a grid pattern, when the part is rotated, the cell block serving as an extraction unit of the image feature amount, and the part Since the cell blocks as the feature quantity extraction units do not overlap with each other, they cannot be compared, and the part feature quantity needs to be recalculated.

[Other Embodiments]
As mentioned above, although embodiment of this invention was described, it cannot be overemphasized that this invention can take a various form, without being limited to the said embodiment.

  (1) In the above embodiment, a correspondence table indicating the relationship between the part rotation amount η and the bin number array is prepared for 360 °, but only a necessary angle is prepared according to actual image distortion. It may be.

  (2) In the above embodiment, the adjustment of the part feature amount according to the part rotation amount η is realized by changing the order of the elements of the multidimensional vector according to the correspondence table with reference to the value when η = 0 °. However, a part feature amount corresponding to the part rotation amount η may be prepared in advance, and the part feature amount may be selected according to the calculated part rotation amount η. In this case, it is possible to further reduce the processing load required for adjusting the part feature amount.

  (3) In the above embodiment, the coordinates of the projection point of the reference point corresponding to the search point are calculated each time. However, the relationship between the search point and the projection point is registered in advance in the lookup table. You may comprise so that it may refer.

  (4) In the above embodiment, the object identification unit is realized by processing (software) executed by the CPU. However, the whole or a part thereof may be realized by hardware such as a logic circuit. Good.

  (5) Each component of the present invention is conceptual and is not limited to the above embodiment. For example, the functions of one component may be distributed to a plurality of components, or the functions of a plurality of components may be integrated into one component. Further, at least a part of the configuration of the above embodiment may be replaced with a known configuration having the same function. In addition, at least a part of the configuration of the above embodiment may be added to or replaced with the configuration of the other embodiment.

DESCRIPTION OF SYMBOLS 1 ... Driving assistance system 2 ... Imaging device 3 ... Image processing part 4 ... Driving assistance execution part 10 ... Object identification part 20 ... Identification information storage part 30 ... Communication part 41 ... Monitor 42 ... Speaker

Claims (7)

Prepared for each object to be detected, each having a plurality of parts corresponding to a part of the object, and representing the reference position of each part and the characteristics of each part representing the relative positional relationship between the parts A target model storage means (20) for storing a target object model in which a part feature amount that is a feature amount is defined;
Image acquisition means (10: S110) for acquiring an image having distortion;
Feature extraction means (10: S120 to S130) for extracting, for each unit area cut out from the image, an image feature quantity that is a feature quantity of the unit area based on the image acquired by the image acquisition means;
A point obtained by projecting a search point set in the three-dimensional space to be imaged onto the image plane is set as a projection search point, and the target model is adjusted according to the degree of image distortion obtained for each projection search point. Model adjusting means (10: S160),
The object associated with the object model is evaluated by evaluating the similarity between the image feature amount extracted by the feature extraction unit and the adjustment model that is the object model adjusted by the model adjustment unit. An object detecting means for detecting (10: S170);
An object detection apparatus comprising: The target model storage means stores at least the target model corresponding to an image without distortion,
The model adjusting unit obtains a positional deviation amount with respect to a case where there is no distortion as an image distortion occurring at the projection search point, and changes a reference position of each part constituting the object model according to the positional deviation amount. The object detection apparatus according to claim 1.   The model adjusting means obtains a rotation amount with respect to a case where there is no distortion as a distortion of an image generated at the projection search point, and changes a part feature amount of each part constituting the object model according to the rotation amount. The object detection device according to claim 2. A plurality of cells arranged rotationally symmetrically with respect to rotation at a predetermined angle are defined as cell blocks, and the luminance gradient obtained for each pixel is voted to one of a plurality of bins each associated with a different gradient direction. Thus, a luminance gradient histogram is generated for each cell, and a multidimensional vector generated by concatenating the luminance gradient histogram in units of the cell blocks is used as the feature amount.
The model adjusting unit changes the part feature amount by changing the order of the elements of the multidimensional vector representing the part feature amount so that the order of bins of the luminance gradient histogram is changed according to the rotation amount. The object detection apparatus according to claim 3, wherein: The target model storage means stores a plurality of part feature amounts corresponding to the rotation amount for each of the parts,
The object detection apparatus according to claim 3, wherein the model adjustment unit selects a part feature amount of each part according to the rotation amount.   The model adjusting unit is configured to calculate distortion of an image generated at the projection search point by geometric calculation based on calibration information of the imaging unit that has generated the image and a three-dimensional positional relationship between the imaging unit and the search point. The object detection device according to claim 1, wherein the object detection device calculates the object detection device.   The object detection apparatus according to claim 1, wherein the image acquisition unit acquires image information from a fisheye camera.