JP2008205644A - Image processing device and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable a driver to intuitively determine a collision risk. <P>SOLUTION: A vision auxiliary system mounted to a vehicle is equipped with an image processing LSI 3. The image processing LSI 3 extracts a motion vector from a photographed image of consecutive two frames or thinned-out two frames on a time axis when a front of the vehicle is photographed by a camera 1 mounted to a side mirror at the front passenger seat side of the vehicle, and qualifies a pixel which has a collision risk based on the direction of the motion vector. Then, the image processing LSI 3 highlights the pixel which has the collision risk in displaying the photographed image photographed by the camera 1. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method in a vehicle vision assistance system.

  There is a system called a side blind monitor as a system for assisting the visual field in the blind spot of an automobile. This system assists the field of view by photographing the front side of the passenger seat of the car, which is the blind spot of the driver, with a camera attached to the side mirror on the passenger side of the car, and displaying the camera image on the monitor inside the car. It is. In such a system, not only the camera image is simply displayed on the monitor, but also the guideline calculated from the shape of the car body is combined with the camera image and displayed on the monitor so that the vehicle sense can be easily grasped. There is also a system like this.

FIG. 5 is a diagram showing an example of the monitor screen of the system. As shown in the figure, the guideline 31 is synthesized and displayed on the camera video.
Various systems have been proposed for performing predetermined processing based on images captured by a camera attached to a vehicle, such as such a visual field assistance system (for example, Patent Documents 1, 2, and 3). .
JP 2001-239832 A JP 2002-314991 A JP 2006-1000096 A

  By the way, in a system in which the above-described guidelines are combined with the camera video and displayed on the monitor, the guidelines are not naturally blended into the camera video and are difficult to judge intuitively. Since the guideline expresses the positional relationship attached to the ground and does not have information on the height direction, it is effective only for visually recognizing the gutter position and for obstacles that grow vertically from the ground, and at higher positions than the ground surface. There is a risk of misidentification of obstacles (such as trees) of complicated shapes that bite into the car. Also, it is effective only when the shooting angle of elevation falls toward the ground, and the guideline does not make sense if you want to check far away and are looking straight ahead.

  In view of the above circumstances, an object of the present invention is to provide an image processing apparatus and an image processing method that enable a driver to intuitively determine a collision risk in a vehicle vision assistance system.

  In order to achieve the above object, an apparatus according to a first aspect of the present invention is an image processing apparatus in a vehicular visibility assisting system, and is attached to the most end portion or near the most end portion on one side in the vehicle width direction of the vehicle. A motion vector extracting means for extracting a motion vector from captured images for a plurality of consecutive frames or a plurality of frames thinned out on a time axis in which the front of the vehicle is photographed by the captured camera, and extracted by the motion vector extracting means A pixel at risk of collision is identified based on the direction of the motion vector.

According to this apparatus, pixels having a risk of collision can be identified based on motion vectors extracted from a plurality of frames of captured images.
According to a second aspect of the present invention, in the first aspect, the apparatus according to the first aspect statistically processes the motion vector extracted by the motion vector extraction unit, and the horizontal component is a left-direction motion vector and the horizontal component is A separation boundary with a motion vector in the right direction is extracted, and the pixel having the risk of collision is identified based on the separation boundary.

According to this apparatus, a pixel having a collision risk can be identified based on a separation boundary between a horizontal motion vector having a horizontal direction and a horizontal component having a horizontal motion vector.
The apparatus according to a third aspect of the present invention is characterized in that, in the first or second aspect, the pixel at risk of collision is highlighted when displaying a photographed image photographed by the camera. .

According to this apparatus, pixels having a risk of collision are highlighted, so that the driver can intuitively determine the risk of collision.
An image processing apparatus according to a fourth aspect of the present invention is the image processing apparatus according to the second aspect, wherein the vehicle is an automobile, and the camera is attached to a side mirror on the passenger seat side of the automobile, The position of the separation boundary is corrected based on the distance in the vehicle width direction from the section to the camera mounting position.

  According to this apparatus, since the position of the separation boundary is corrected, the accuracy of the position of the separation boundary is further improved, and the accuracy of the pixel having a collision risk recognized according to the separation boundary is further improved.

  In addition, the present invention is not limited to the apparatus according to each aspect described above, and can be configured as a method.

  According to the present invention, the driver can intuitively determine the collision risk.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram of a visual field assistance system provided with an image processing apparatus according to an embodiment of the present invention. In the present embodiment, it is assumed that the visibility assisting system is a side blind monitor system, and this system is mounted on a right-hand drive type vehicle traveling on a left-handed road.

  As shown in the figure, the visual field assistance system includes a camera 1, a frame buffer 2, an image processing LSI (Large Scale Integration) 3, a display 4, and the like as main components.

  The camera 1 is, for example, a CCD (Charge Coupled Devices) camera or the like, which is attached to a side mirror (for example, a door mirror or a rearview mirror) on the passenger seat side of the automobile, and is located in front of the passenger seat side of the automobile which is a blind spot of the driver. Shoot at a frame rate of.

  The frame buffer 2 has a rear frame region 5 and a front frame region 6 and stores image data of captured images of two frames that are consecutively taken or thinned out on the time axis captured by the camera 1. The rear frame area 5 is an area for sequentially storing image data of captured images captured by the camera 1 for each frame. The previous frame area 6 is an area for storing the image data stored in the subsequent frame area 5 in order or one for every predetermined number of frames. As a result, the previous frame area 6 stores the image data of the previous frame in time with respect to the image data stored in the subsequent frame area 5.

The image processing LSI 3 is an example of an image processing apparatus according to the present embodiment, and includes a motion vector horizontal component extractor 7, a motion vector direction determiner 8, a left score buffer 9, and a right score buffer 10.
, A watershed score line buffer 11, a magnitude comparator 12, a maximum value pixel detector 13, a straight / left turn / right turn determination unit 14, a collision risk pixel enhancement display effector 15, and a video display 16.

  The motion vector horizontal component extractor 7 extracts the motion vector of each pixel from the image data of the captured image for two frames stored in the rear frame region 5 and the previous frame region 6, and further, for each pixel. Extract the horizontal component of the motion vector.

  In this embodiment, since a motion vector is extracted for each pixel, the motion vector can also be referred to as a pixel flow. In this embodiment, the horizontal direction of the captured image is the X direction, the vertical direction is the Y direction, the left direction in the horizontal direction is the -X direction, and the right direction is the + X direction.

  For each horizontal component of the motion vector of each pixel extracted by the motion vector horizontal component extractor 7, the motion vector direction determiner 8 determines whether the horizontal component of the motion vector is the left direction, the right direction, It is determined whether it is 0 (no horizontal component).

In the left score buffer 9, the number when the horizontal component of the motion vector is determined to be in the left direction by the motion vector direction determiner 8 is stored as a score.
In the right score buffer 10, the number when the horizontal component of the motion vector is determined to be in the right direction by the motion vector direction determining unit 8 is stored as a score.

  The watershed score line buffer 11 is a line buffer corresponding to the pixel position in the horizontal direction of the captured image (X coordinate of the captured image), and the motion vector direction determiner 8 determines that the horizontal component of the motion vector is zero. Is stored as a score for each pixel position in the horizontal direction. For example, when the number of pixels in the horizontal direction of the captured image is 640, 640 scores corresponding to the respective pixel positions in the horizontal direction are stored in the watershed score line buffer 11.

The magnitude comparator 12 compares the scores stored in the left score buffer 9 and the right score buffer 10.
The maximum value pixel detector 13 detects the pixel position (X coordinate) of the maximum score from the score of each pixel position in the horizontal direction stored in the watershed score buffer 11.

  The straight / left turn / right turn determination unit 14 determines whether the vehicle is moving straight, turning left, or turning right based on the comparison result of the magnitude comparator 13 and the detection result of the maximum value pixel detector 13. It is determined whether it is.

  The collision risk pixel highlighting effector 15 recognizes a collision risk pixel (a pixel having a collision risk) from the determination result of the straight / left turn / right turn determination unit 14 and the detection result of the maximum value pixel detector 13, The video display 16 is instructed to highlight this.

  The video display 16 displays on the display 4 a captured image in which the collision risk pixel is highlighted based on the image data of the captured image stored in the rear frame region 5 and the instruction of the collision risk pixel enhancement display effector 15. Let However, when the collision risk pixel highlighting effector 15 does not recognize the collision risk pixel, the captured image is normally displayed on the display 4 based on the image data stored in the rear frame region 5.

Note that a captured image in which the collision risk pixels are highlighted can be generated using a technique such as α blend.
The display 4 is a liquid crystal display or the like, for example, and is attached at a position in the vehicle that can be visually recognized by the driver.

Next, the operation of this visibility assisting system will be described focusing on the processing of the image processing LSI 3.
FIG. 2 is a flowchart showing processing in units of frames of the image processing LSI 3 in this visual field assistance system. Note that this processing is performed when the automobile is moving, and shooting with the camera 1 is started, and image data of the shot image is stored in the rear frame area 5 and the front frame area 6. Shall be done from time to time.

  As shown in the figure, in this process, first, the motion vector of each pixel is extracted from the image data of the captured images for two frames stored in the rear frame area 5 and the front frame area 6 (S1). .

  Subsequently, the processing target scanning line is the first scanning line and the processing target pixel is the first pixel, and a horizontal component is extracted from the motion vector extracted in S1 for the processing target pixel in the processing target scanning line (S2). .

Subsequently, it is determined whether the direction of the horizontal component extracted in S2 is a negative direction (left direction), 0, or a positive direction (right direction) (S3).
If the determination result in S3 is negative, the score stored in the left score buffer 9 is increased by 1 (S4).

  In the present embodiment, all the scores stored in the left score buffer 9, the right score buffer 10, and the watershed score line buffer 11 are all reset to 0 before the start of the processing of S1.

  If the determination result in S3 is 0, the corresponding score stored in the watershed score line buffer 11 is increased by 1 (S5). For example, when the processing target pixel is the first pixel, the score corresponding to the first pixel stored in the watershed score line buffer 11 is increased by one.

If the determination result in S3 is positive, the score stored in the right score buffer 10 is increased by 1 (S6).
Subsequently, it is determined whether or not all pixels in the processing target scanning line have been processed as processing target pixels (S7). If the determination result is No, the processing target pixel is moved to the next pixel ( S8), return to S2 If the determination result of S7 is Yes, it is determined whether or not all scanning lines have been processed as processing target scanning lines (S9), and if the determination result is No, the processing target The scanning line is moved to the next scanning line (S10), and the process returns to S2.

When the determination result of S9 is Yes, the pixel position of the maximum score is searched from the score of each pixel position of the horizontal component stored in the watershed score line buffer 11 (S11).
Subsequently, it is determined whether or not the maximum score in the score of each pixel position stored in the watershed score line buffer 11 is greater than 0 (S12).

If the determination result in S12 is Yes, it is determined that the vehicle is traveling straight, and the coordinates (X coordinate) of the pixel position searched in S11 are recognized as a collision risk boundary (S13). The collision risk boundary is a separation boundary between the horizontal motion vector having the left direction and the horizontal component having the right direction. In the present embodiment, the collision risk boundary, which is this separation boundary, is obtained by statistical processing of motion vectors as described above, so that it is locally determined by a moving object (for example, a pedestrian or another vehicle crossing the front). It is possible to eliminate the influence of disturbance of a simple motion vector.

Then, the pixel on the right side from the collision risk boundary is recognized as a collision risk pixel, and a captured image in which the risk pixel is highlighted is displayed on the display 4 (S14).
FIG. 3 is a diagram illustrating an example of a captured image displayed at this time. As shown in the figure, the area on the right side of the recognized collision risk boundary (shaded area in the figure) is recognized as a collision risk pixel and is displayed with a risk emphasis. In addition, the area | region colored in gray of the figure shows the area | region of the tree from which the horizontal component extracted the motion vector rightward. However, this gray colored area is colored for convenience of explanation, and this area is also displayed with risk emphasis on the actual display screen.

  With such a display, the driver can intuitively determine that there is a collision risk in the risk highlighted area. In the example shown in FIG. 3, the driver can easily determine that there is a possibility of colliding with a tree in the area where risk is highlighted. The area on the left side from the collision risk boundary is an area where a sufficient clearance from the automobile is maintained.

  On the other hand, when the determination result of S12 is No, the left score buffer 9 and the right score buffer 10 are referred to (S15), and the score stored in the left score buffer 9 is stored in the right score buffer 10 It is determined whether it is smaller than (S16).

  If the determination result in S16 is Yes, it is determined that the vehicle is turning left, and the entire screen of the display 4 is recognized as a collision risk region (S17). That is, all pixels are recognized as collision risk pixels. Then, the entire screen of the display 4 is highlighted (S18). That is, a captured image in which all the pixels are risk-highlighted as collision risk pixels is displayed on the display 4. As a result, the driver can intuitively determine that there is a collision risk.

  If the determination result in S16 is No, it is determined that the vehicle is turning right, the entire screen is recognized outside the collision risk area (S19), and the entire screen of the display 4 is normally displayed (S20). That is, the captured image is normally displayed on the display 4.

By repeatedly performing the above processing in units of frames, the driver can intuitively determine the collision risk while the vehicle is running.
The above processing is based on the following principle.

  FIG. 4 is a schematic diagram of motion vectors extracted from image data for a plurality of different frames on the time axis. In this embodiment, since the outside world is stationary and the camera (own vehicle) side moves, the entire screen flows from a distant view to a foreground, and a radial motion vector is obtained as shown in FIG. Example when the side is going straight). Therefore, the right vector (horizontal component is the motion vector in the right direction) and the left vector (horizontal component is the motion vector in the left direction) are separated like a watershed at a certain position on the screen (see the boundary line 21 in the figure). . Since the basic motion vector is radial, the disturbance of the motion vector generated by a pedestrian or another vehicle crossing the front is local (see the dotted line in the figure). Therefore, if the left and right vector separation boundaries are extracted statistically (filtering, averaging, histogram, etc.), the sensitivity to disturbance can be lowered. In the case of the figure, the area on the right side of the boundary line 21 is an area where there is a risk of contact, and the area on the left side of the boundary line 21 is an area with sufficient clearance. For example, if there is an obstacle in the area on the right side of the boundary line 21, there is a risk of contact with the obstacle, and if there is an obstacle in the area on the left side of the boundary line 21, between the obstacle Therefore, a sufficient clearance is maintained.

As described above, according to the present embodiment, the driver who views the screen of the display 4 can intuitively determine the collision risk.
In this embodiment, as shown in S12 and S13 of the flowchart shown in FIG. 2, when the maximum score stored in the watershed score line buffer 11 is larger than 0, the pixel position of the maximum score collides. It is recognized as a risk boundary. However, in consideration of the influence of the above-described motion vector disturbance, the collision risk boundary can be recognized when the maximum score is larger than a predetermined value of 1 or more (for example, 10).

  In this embodiment, the motion vector is extracted from the image data of the captured images for two frames that are continuous or thinned on the time axis. However, the image is captured for three or more frames that are continuous or thinned on the time axis. It can also be set as the structure which extracts a motion vector from the image data of an image.

In the present embodiment, processing is performed by extracting a motion vector. However, a configuration in which an optical flow is extracted instead of a motion vector and processing is performed in the same manner may be employed.
In this embodiment, a motion vector is extracted for each pixel. However, a configuration in which a motion vector is extracted for each pixel block including a plurality of pixels and processing is performed in a similar manner may be employed. However, in this case, it is necessary to determine the number of pixels constituting one pixel block to such an extent that the accuracy of the collision risk boundary does not decrease.

  Further, in the present embodiment, the position of the collision risk boundary can be corrected by the distance in the vehicle width direction from the camera mounting position of the side mirror to the tip of the side mirror, and the accuracy of the collision risk boundary can be further improved. For example, when the vehicle width direction distance from the camera mounting position to the side mirror tip is several centimeters, in the example shown in FIG. 3, the collision risk boundary is on the left side from the current position by the amount corresponding to the centimeters. It is corrected to the position. Thereby, since the accuracy of the collision risk boundary is further improved, it is possible to further improve the accuracy of the region having the contact risk and the region where the clearance is maintained.

  In the present embodiment, the visibility assisting system shown in FIG. 1 is mounted on a right-hand drive vehicle that travels on a left-handed road. For example, a left-hand drive vehicle that travels on a right-handed road Of course, it is also possible to mount it on.

Further, in the present embodiment, the example in which the automobile mounts the visibility assisting system shown in FIG. 1 has been described, but it is of course possible to mount a vehicle other than the automobile.
Although the present invention has been described in detail above, the present invention is not limited to the above-described embodiment, and it is needless to say that various improvements and changes may be made without departing from the gist of the present invention.

[BRIEF DESCRIPTION OF THE DRAWINGS] It is a block diagram of the visual field assistance system provided with the image processing apparatus based on one embodiment of this invention. It is a flowchart which shows the process per frame of the image processing LSI in a visual field assistance system. It is an example of a display screen when a collision risk pixel is highlighted. It is a schematic diagram of the motion vector extracted from the image data for different frames on the time axis. It is a figure which shows an example of the monitor screen in the conventional visual field assistance system.

Explanation of symbols

1 Camera 2 Frame buffer 3 Image processing LSI
4 Display 5 Rear frame region 6 Previous frame region 7 Motion vector horizontal component extractor 8 Motion vector direction determiner 9 Left score buffer 10 Right score buffer 11 Divided water scoring line buffer 12 Size comparator 13 Maximum pixel detector 14 Straight / left Turn / Right turn discriminator 15 Collision risk pixel highlighting effector 16 Video display 21 Collision risk boundary 31 Guidelines

Claims (5)

An image processing apparatus in a vehicle vision assistance system,
A motion vector is obtained from captured images of a plurality of frames that are consecutively thinned or thinned on a time axis in which the front of the vehicle is photographed by a camera attached to one end of the vehicle in the vehicle width direction or near the end. A motion vector extracting means for extracting,
Qualifying pixels at risk of collision based on the direction of the motion vector extracted by the motion vector extraction means;
An image processing apparatus. The motion vector extracted by the motion vector extraction means is statistically processed to extract a separation boundary between a horizontal motion vector that is a leftward motion vector and a horizontal component that is a rightward motion vector, and based on the separation boundary Certify pixels at risk of collision,
The image processing apparatus according to claim 1. Highlighting the pixels at risk of collision when displaying captured images captured by the camera,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The vehicle is an automobile;
The camera is attached to a side mirror on the passenger seat side of the car,
Correcting the position of the separation boundary based on the distance in the vehicle width direction from the end of the side mirror to the mounting position of the camera;
The image processing apparatus according to claim 2. An image processing method in a vehicle vision assistance system,
A motion vector is obtained from captured images of a plurality of frames that are consecutively thinned or thinned on a time axis in which the front of the vehicle is photographed by a camera attached to one end of the vehicle in the vehicle width direction or near the end. Extract and
Qualifying pixels at risk of collision based on the direction of the motion vector;
An image processing method.