JP2005316607A - Image processor and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor and image processing method capable of detecting only a moving object necessary for a driver with a light processing load. <P>SOLUTION: The image processor 1 for detecting the moving object such as a pedestrian is provided with an imaging means 2 for imaging a front of a vehicle, roadway detecting means 11, 12 for detecting a roadway from an imaged image imaged by the imaging means 2, area determining means 13, 14 for determining an existing area where the moving object necessary for the driver possibly exists in the imaged image on the basis of the roadway detected by the roadway detecting means 11, 12, a rectangular forming means 15 for forming a rectangular from the existing area determined by the area determining means 13, 14, and a searching means 16 for searching the moving object to the existing area formed in the rectangular by the rectangular forming means 15. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and an image processing method for detecting a moving object from a captured image.

In recent years, various devices for assisting the driver have been developed, and some of these devices provide the driver with information on a pedestrian in front of the vehicle. As a method of detecting a pedestrian, for example, there is a method of capturing an image of the front of the vehicle with a camera and detecting the pedestrian from the captured image by template matching or the like. Furthermore, in the moving object detection device described in Patent Document 1, the distance distribution from the vehicle in the captured image is estimated, and the moving object (pedestrian) is detected using a template having a size corresponding to the distance. ing.
JP 2002-99997 A

  When providing pedestrian information to the driver, only the pedestrian information that may jump out of the roadway is required for the driver. Therefore, since information on pedestrians far away from the roadway is not necessary, it is necessary to detect only pedestrians near the roadway. However, in the conventional pedestrian detection, since all pedestrians present in the captured image are detected, pedestrians far away from the roadway are also detected. Therefore, a separate process for determining whether or not the detected pedestrian is a pedestrian that needs to be provided as information for the driver is required. Furthermore, in the conventional pedestrian detection, in order to detect all the pedestrians existing in the captured image, matching must be performed while changing the size of the template, so the pattern for performing the matching becomes enormous. As a result, the processing load increases.

  Therefore, an object of the present invention is to provide an image processing apparatus and an image processing method capable of detecting only a moving object necessary for a driver with a light processing load.

  An image processing apparatus according to the present invention includes: an imaging unit that images a vehicle front; a roadway detection unit that detects a roadway from an image captured by the imaging unit; and a driver in an image captured based on the roadway detected by the roadway detection unit. Area determining means for determining an existence area in which there may be a moving object necessary for the object, a rectangularizing means for rectangularizing the existence area determined by the area determining means, and an existence area rectangularized by the rectangularizing means And searching means for searching for a moving object.

  In this image processing apparatus, the front of the vehicle is imaged by the imaging means, and the roadway is detected from the captured image by the roadway detection means. Usually, an area for moving animals (pedestrians, etc.) other than vehicles such as sidewalks and roadside belts is provided outside the roadway. In view of this, the image processing apparatus determines an existing area where a driver may need a moving object from the captured image based on the roadway detected by the area determining means. Since the existence area is an area cut out from the captured image, the size of the moving object on the image changes depending on the position in the forward direction of the vehicle (that is, the depth direction of the captured image). For example, when there are pedestrians having the same height at positions close to and far from the front of the vehicle in the presence area, the pedestrians at far positions are smaller than those at the close positions on the image of the presence area. Therefore, in the image processing apparatus, the existence area is rectangularized by the rectangularization means. In this way, by making the existence area rectangular, the height is constant regardless of the position in the front direction of the vehicle, and the moving object on the image has the same size reference. Therefore, in the case of the rectangular presence area, pedestrians having similar heights at positions close to and far from the front of the vehicle have the same size on the image. In view of this, the image processing apparatus searches for a moving object from the existence area rectangularized by the search means. In the rectangular presence area, the standard of size is unified regardless of the position in the front direction of the vehicle. Therefore, when searching for a certain size animal body (for example, an adult pedestrian), a certain size is required. It is only necessary to search for the animal body. That is, in the rectangular existence area, the search size is determined to be a predetermined size in accordance with the type of moving object to be searched, and only the determined size need be searched. In addition, in the rectangular existence region, the size of the moving object on the image becomes smaller as the position is farther from the roadway. Therefore, in the image processing apparatus, it is possible to automatically exclude moving objects far away from the roadway by searching only for moving objects corresponding to the size located in the vicinity of the roadway. Thus, in this image processing apparatus, the processing load is light and only the moving object necessary for the driver can be detected.

  In addition, as a moving body, it is a pedestrian, a bicycle, a wheelchair, and an animal, for example. The animal body necessary for the driver is an animal body that can go on the roadway.

  The roadway detection means of the image processing apparatus of the present invention may be configured to detect a road boundary line.

  In this image processing apparatus, a road boundary line such as a lane is detected by the roadway detection means, and the roadway is specified. As described above, the image processing apparatus can easily detect the roadway by the road boundary line. Examples of the road boundary line include a lane (white line or the like), a road shoulder, a block or a step that partitions the sidewalk and the roadway.

  An image processing method according to the present invention is an image processing method for detecting a moving object from a captured image obtained by imaging the front of a vehicle, based on a roadway detection step for detecting a roadway from the captured image, and a roadway detected in the roadway detection step. In the captured image, an area determination step for determining an existence area where a moving object necessary for the driver may exist, a rectangularization step for rectangularizing the existence area determined in the area determination step, and a rectangle in the rectangularization step And a search step of searching for a moving object with respect to the existence region that has been converted into a feature. Furthermore, a road boundary line may be detected in the road detection step of the image processing method of the present invention.

  Each of the image processing apparatuses described above has the same operational effects as the image processing apparatus.

  According to the present invention, only a moving object necessary for the driver can be detected from a captured image with a light processing load.

  Embodiments of an image processing apparatus and an image processing method according to the present invention will be described below with reference to the drawings.

  In the present embodiment, the present invention is applied to a occupant detection device mounted on an automobile. The pedestrian detection device according to the present embodiment detects only pedestrians present in the vicinity of the roadway. In the pedestrian detection device according to the present embodiment, the boundary of the roadway is specified by detecting the white line indicating the lane, and the pedestrian detection area (existing area) is determined from the white line. In the pedestrian detection device according to the present embodiment, the pedestrian detection area is converted into a rectangular area, and template matching is performed on the matching area cut out from the rectangular area.

  With reference to FIG.1 and FIG.2, the structure of the pedestrian detection apparatus 1 is demonstrated. FIG. 1 is a configuration diagram of a pedestrian detection device according to the present embodiment. FIG. 2 is a diagram illustrating a process in each unit in the image ECU of FIG. 1, (a) is an image showing a white line determined by a white line determination unit, and (b) is a vanishing point calculated by a vanishing point calculation unit. (C) is an image showing a pedestrian detection region determined by the pedestrian detection region determination unit, (d) is an image showing a rectangular region converted by the rectangular region conversion unit, e) is an image showing the matching area cut out from the rectangular area by the pedestrian determination unit, and (f) is an image showing the pedestrian determined by the pedestrian determination unit.

  The pedestrian detection device 1 images the front of the automobile and detects a pedestrian necessary for the driver from the captured image. The detected pedestrian information is provided to the driver by an alarm, a voice output, a screen display, etc. for notifying that a pedestrian is present in the vicinity of the roadway. In particular, the pedestrian detection apparatus 1 can detect only pedestrians necessary for the driver without detecting all pedestrians present in the captured image, and the processing load is light. For this purpose, the pedestrian detection apparatus 1 includes a CCD [Charge Coupled Device] camera 2 and an image ECU [Electronic Control Unit] 3. The image ECU 3 includes an image input unit 10, an edge detection unit 11, a white line determination unit 12, A vanishing point calculation unit 13, a pedestrian detection region determination unit 14, a rectangular region conversion unit 15, and a pedestrian determination unit 16 are configured.

  In this embodiment, the CCD camera 2 corresponds to the imaging means described in the claims, the edge detection unit 11 and the white line determination unit 12 correspond to the roadway detection means described in the claims, and disappears. The point calculation unit 13 and the pedestrian detection region determination unit 14 correspond to the region determination unit described in the claims, and the rectangular region conversion unit 15 corresponds to the rectangularization unit described in the claims, and the pedestrian determination The unit 16 corresponds to search means described in the claims.

  The CCD camera 2 is attached in front of an automobile on which the pedestrian detection device 1 is mounted. The CCD camera 2 captures the front of the automobile and obtains the captured color image (for example, an image by RGB [Red Green Blue]). The CCD camera 2 transmits the captured image data to the image ECU 3. The CCD camera 2 has a wide imaging range in the left-right direction, and can capture images such as sidewalks and roadside zones provided outside the roadway.

  The image ECU 3 includes a CPU [Central Processing Unit], a ROM [Read Only Memory], a RAM [Random Access Memory], and the like, and each processing unit is configured. The image ECU 3 takes in captured image data from the CCD camera 2 and detects only pedestrians within several tens of meters ahead of the vehicle and in the vicinity of the road from the captured image. The image ECU 3 transmits the pedestrian information to an alarm device (not shown) or the like in order to inform the driver of the detected pedestrian information.

  The image input unit 10 takes captured image data from the CCD camera 2 and stores the captured image data in a predetermined recording area.

  The edge detection unit 11 detects an edge indicating a contour such as a lane (white line) from the captured image, and generates an edge image. As a method for extracting an edge, a conventional edge extraction method (for example, a Sobel filter) is used. In particular, when only the lane is detected, the lane is white and painted on the road. Therefore, the edge detection may be performed after extracting the white portion from the captured image. When extracting the white part, the RGB captured image is converted into an image having hue, saturation, and lightness component information using HSI (Hue Saturation Intensity) conversion, and the white part is extracted from these component information images.

  The white line determination unit 12 extracts a straight line from the edge image by Hough transform. And the white line determination part 12 determines whether the extracted straight line is a white line, and detects the white line of the both right and left which shows the boundary of a roadway. As the white lines, for example, as shown in FIG. 2A, white lines WL and WR on both the left and right sides of the roadway on one side are detected.

  The vanishing point calculation unit 13 obtains primary expressions representing the detected white lines on both the left and right sides based on predetermined points (for example, the center point and the lower left point) of the captured image. Then, the vanishing point calculation unit 13 extends the white lines on the left and right sides in the depth direction in the captured image using each of the linear expressions, and calculates the intersection. This intersection is a vanishing point, and is a point where the roadway disappears on the captured image. As the vanishing point, for example, as shown in FIG. 2B, the vanishing point DP that is the intersection of the extended lines of the white lines WL and WR is calculated.

  The pedestrian detection area determination unit 14 determines a pedestrian detection area (ABCD) based on the vanishing point for the captured image. The pedestrian detection area is an area where there is a possibility that a pedestrian necessary for the driver exists in the captured image, and an area including a sidewalk, a roadside belt, etc. provided outside the roadway (especially the left outside) in the captured image. It is. A pedestrian necessary for the driver is a pedestrian who may jump out onto the roadway and is located near the roadway. Even if it is located near the roadway, pedestrians far away from the host vehicle in the forward direction are information that is not necessary for the driver until the host vehicle is approached to some extent. Need to be excluded from. In addition, since the pedestrian detection area needs to have a height that sufficiently includes the pedestrian, even if a tall person stands at the nearest position from the roadway in the captured image, the person is sufficiently It is set as the area | region which has the height (for example, about 2 m) including. Accordingly, the pedestrian detection area is located outside the roadway in the captured image, and a distance of about several tens of meters is set in the forward direction from the host vehicle in the depth direction, and a tall person is sufficiently provided in the height direction. Contains height is set. How far the depth direction is set may be a fixed distance set in advance, or a distance that varies according to the vehicle speed (longer the higher the vehicle speed).

  On a relatively thin road with no center line, there is a possibility that a pedestrian located on the sidewalk or roadside belt on the right outer side of the road may jump out to the front of the vehicle. On the other hand, on a wide road with a center line and a median strip, there is almost no possibility that a pedestrian located on a sidewalk or a roadside belt on the right outer side of the road will jump out to the front of the host vehicle. Therefore, it may be determined whether the pedestrian detection area is set on both the left and right sides or only on the left side depending on the presence or absence of the center line or the center separation band.

  In the pedestrian detection area determination unit 14, first, the point closest to the detected white line is set as the first end point A of the pedestrian detection area. Then, the pedestrian detection area determination unit 14 sets the point separated from the first end point A on the white line by the distance set in the depth direction as the second end point B. Further, in the pedestrian detection area determination unit 14, the height in the vertical direction with respect to the white line from the intermediate point between the first end point A and the second end point B on the white line according to the height set in the height direction. A separated point E is set, and a line is drawn connecting the set point E and the vanishing point. In the pedestrian detection area determination unit 14, an intersection point between a line drawn from the vanishing point and a line drawn from the second end point B in a direction perpendicular to the white line is set as a third end point C. Further, in the pedestrian detection area determination unit 14, a line drawn from the vanishing point is extended to the near side, and a point on the most front side of the extension line is set as a fourth end point D. A trapezoidal area formed by these four end points A, B, C, and D is defined as a pedestrian detection area. As the pedestrian detection area (ABCD), for example, as shown in FIG. 2C, the pedestrian detection area DA is determined based on the white line WL.

  By the way, when obtaining a more accurate pedestrian detection area, according to the height set in the height direction, from each point on the white line from the first end point A to the second end point B to the white line Set the points separated in height in the vertical direction, and draw a line connecting the set points. Then, a region sandwiched between the white line from the first end point A to the second end point B and a line connecting the points is set as a pedestrian detection region. In the case of this pedestrian detection area, the third end point C to the fourth end point D is not a straight line but a curved line, and becomes wider toward the front side. In this method of obtaining the pedestrian detection area, it is not necessary to obtain the vanishing point.

  The rectangular area conversion unit 15 cuts out a pedestrian detection area (ABCD) from the captured image, and converts the pedestrian detection area into a rectangular area (abcd). In the pedestrian detection area, the size of the pedestrian of the same size (height) actually changes depending on the position of the host vehicle in the forward direction. Therefore, the pedestrian detection area is rectangularized in order to match the size reference on the image regardless of the position of the host vehicle in the forward direction. The rectangular area is an area in which the height of the pedestrian detection area is constant, and is rectangular.

  In the rectangular area conversion unit 15, with the pedestrian detection area near the center in the depth direction as a reference, the image on the back side with respect to the image of the pedestrian detection area is enlarged and the front side with respect to the reference is reduced to reduce the pedestrian. Make the detection area rectangular. As the rectangular area (abcd), a rectangular area RA having a constant height is formed as shown in FIG. As shown in FIG. 2 (c), in the pedestrian detection area DA, pedestrians X and Y having similar heights exist in the vicinity from the roadway, but the pedestrian Y is far away in the forward direction of the host vehicle. Therefore, it is considerably smaller than the pedestrian X on the image. However, in the rectangular area RA, the pedestrian X and the pedestrian Y have the same size.

  The pedestrian determination unit 16 sequentially cuts out a certain matching region (stub) from the rectangular region (abcd). The matching area is an area to be matched with the template when performing template matching, and is a rectangular area (for example, a height of 1.6 to 1.8 m) that matches the size of an adult on the image of the rectangular area (abcd). Territory that includes other people). As the matching area (stub), for example, as shown in FIG. 2 (e), a matching area MA that just includes pedestrians X and Y located in the vicinity of the roadway is cut out. Incidentally, the pedestrian Z located far away from the roadway is considerably smaller than the matching area MA on the image.

  The pedestrian determination unit 16 performs matching with a pedestrian template for each extracted image of the matching area, and calculates a correlation value. This template is a template having the same size as the size of the pedestrian on the image set in the matching area, and the correlation value with the pedestrian just included in the matching area is high. Furthermore, the pedestrian determination unit 16 determines whether or not the correlation value is equal to or greater than a threshold value. If the correlation value is equal to or greater than the threshold value, the pedestrian determination unit 16 determines that the pedestrian is necessary for the driver. The pedestrian determination unit 16 acquires position information and the like for the own vehicle for the determined pedestrian as necessary. As pedestrians necessary for the driver, for example, as shown in FIG. 2 (f), only pedestrians X and Y located in the vicinity of the roadway from the rectangular area RA are determined. Incidentally, the pedestrian Z is smaller than the matching area MA and has a low correlation value with the template, and is not determined as a pedestrian necessary for the driver.

  When detecting a child as a pedestrian, if the matching area and template are simply reduced according to the size of the child, an adult far away from the roadway located on the far side in the rectangular area is also detected as a pedestrian. End up. In the rectangular area, the farthest distance from the roadway, the higher the position of the lowermost end of the pedestrian. Therefore, when the matching area is cut out from the rectangular area, a cutting range from the lower end of the rectangular area to a predetermined height is provided, and the matching area whose lower end is included in the cutting range is cut out. The cut-out range is set based on a range in the vicinity from the roadway where a pedestrian necessary for the driver may exist. In this way, by reducing the size of the matching region (pedestrian template) and limiting the height position at which the matching region is cut out, it is possible to detect a child as a pedestrian. Incidentally, even when a child is detected as a pedestrian, a restriction may be imposed on the height position at which the matching area is cut out. By adding a restriction in this way, the number of matching regions to be cut out is reduced, and the processing load is further reduced.

  With reference to FIG. 1, operation | movement of the pedestrian detection apparatus 1 is demonstrated. In particular, the processing of the image ECU 3 will be described along the flowchart of FIG. FIG. 3 is a flowchart showing a flow of processing in the image ECU of FIG.

  In the pedestrian detection device 1, the CCD camera 2 images the front of the host vehicle at regular intervals. And in the pedestrian detection apparatus 1, the captured image is taken into image ECU3 for every fixed time, and is hold | maintained in image ECU3.

  The image ECU 3 detects an edge from the captured image and generates an edge image (S1). Then, the image ECU 3 extracts white lines on the left and right sides of the roadway from the edge image using the Hough transform (S2).

  In the image ECU 3, the white lines on both the left and right sides are extended in the depth direction to determine the vanishing point of the roadway (S3). Subsequently, the image ECU 3 obtains a pedestrian detection area (ABCD) where there is a possibility that a pedestrian necessary for the driver exists in the captured image with the vanishing point as a reference (S4). The pedestrian detection area (ABCD) does not include pedestrians far away from the host vehicle in the forward direction. Further, in the image ECU 3, a pedestrian detection area (ABCD) is cut out from the captured image, and the pedestrian detection area (ABCD) whose height from the white line changes at the position in the depth direction is changed to a rectangular area (abcd) having a constant height. Conversion is performed (S5).

  The image ECU 3 sequentially cuts out a matching area (stub) having a certain size according to the size of the pedestrian from the rectangular area (abcd) (S6). Then, the image ECU 3 performs template matching for each extracted matching area, and determines whether the driver is a pedestrian necessary for the driver (S7). At this time, in template matching, matching is performed using only a template having a size corresponding to the matching region (stub), and a pedestrian (a pedestrian far from the roadway) that does not match the size of the matching region (stub). ). In the pedestrian detection device 1, when a pedestrian necessary for the driver is detected, information on the pedestrian is transmitted to an alarm device or the like.

  In this pedestrian detection device 1, by converting the pedestrian detection area into a rectangular area having a constant height, the size of the pedestrian on the image does not change depending on the position in the front direction of the host vehicle. Furthermore, in the pedestrian detection device 1, a matching area having a certain size is set, and template matching is performed only with a template that matches the size of the matching area, so that a pedestrian having a certain size in the rectangular area is obtained. Can be detected. Thereby, a pedestrian far away from the roadway can be automatically excluded, and only a pedestrian existing in the vicinity of the roadway can be detected.

  As described above, the pedestrian detection apparatus 1 can detect only the pedestrians necessary for the driver without detecting all the pedestrians present in the captured image. Moreover, in the pedestrian detection apparatus 1, since it is not necessary to perform matching by changing the size of the template, the processing load is light and the calculation cost can be greatly reduced. Furthermore, in the pedestrian detection apparatus 1, the vanishing point of a roadway can be calculated | required from the white line of both right and left sides, and a pedestrian detection area | region can be calculated | required easily on the basis of a vanishing point.

  As mentioned above, although embodiment which concerns on this invention was described, this invention is implemented in various forms, without being limited to the said embodiment.

  For example, in this embodiment, the present invention is applied when detecting a pedestrian, but it can also be applied when detecting other moving objects such as bicycles and animals. When detecting other moving objects, it is necessary to set the existence area and the matching area according to the size of the moving object to be detected.

  In the present embodiment, the roadway is detected by detecting the white line indicating the lane, but the roadway may be detected by detecting a road, a block or a step that separates the lane and the sidewalk. Alternatively, the road area may be detected using the color of the road. In the white line detection, the white line is detected from the edge image by the Hough transform. However, the white line may be detected by using a stereo view using a stereo camera.

  In the present embodiment, it is configured to determine whether or not a pedestrian by template matching, but may be configured to detect pedestrians by other methods. In addition, when a strong classifier such as SVM [Support Vector Machine] is used as template matching, the classifier is weak against changes in the size of the template, but this pedestrian detection device changes the size of the template. It is applicable because it does not.

It is a block diagram of the pedestrian detection apparatus which concerns on this Embodiment. It is a figure which shows the process in each part in image ECU of FIG. 1, (a) is an image which shows the white line determined by the white line determination part, (b) is an image which shows the vanishing point calculated by the vanishing point calculation part. (C) is an image showing a pedestrian detection area determined by the pedestrian detection area determination unit, (d) is an image showing a rectangular area converted by the rectangular area conversion unit, and (e) is a walking It is an image which shows the matching area | region cut out from the rectangular area | region by the pedestrian determination part, (f) is an image which shows the pedestrian determined by the pedestrian determination part. It is a flowchart which shows the flow of a process in image ECU of FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Pedestrian detection apparatus, 2 ... CCD camera, 3 ... Image ECU, 10 ... Image input part, 11 ... Edge detection part, 12 ... White line determination part, 13 ... Vanishing point calculation part, 14 ... Pedestrian detection area determination part , 15 ... rectangular area conversion unit, 16 ... pedestrian determination unit

Claims (4)

Imaging means for imaging the front of the vehicle;
Roadway detection means for detecting a roadway from a captured image captured by the imaging means;
An area determining means for determining an existing area in which a moving object necessary for the driver may be present in the captured image based on the roadway detected by the roadway detecting means;
A rectangularizing means for rectangularizing the existing area determined by the area determining means;
An image processing apparatus comprising: search means for searching for a moving object with respect to the existence area rectangularized by the rectangularization means.   The image processing apparatus according to claim 1, wherein the roadway detection unit detects a road boundary line. An image processing method for detecting a moving object from a captured image obtained by imaging the front of a vehicle,
A roadway detection step for detecting the roadway from the captured image;
An area determining step for determining an existing area in which a moving object necessary for the driver may exist in the captured image based on the roadway detected in the roadway detecting step;
A rectangularization step of rectangularizing the existence region determined in the region determination step;
An image processing method comprising: a search step of searching for a moving object with respect to the existence region rectangularized in the rectangularization step.   The image processing method according to claim 3, wherein in the roadway detection step, a road boundary line is detected.

Images