JP2008103839A - Apparatus and method for monitoring vehicle - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video image for urging a driver to pay attention to an object in the rear of a vehicle. <P>SOLUTION: A video processing apparatus 102 executes video processing to a video image of the rear of the vehicle which has been picked up with a rear camera 101 to decrease attention-grabbing performance of a region including the object having low necessity of the driver's attention. The video image after video-processing is displayed on a display apparatus 103. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a vehicle monitoring device and a vehicle monitoring method for a driver to monitor a situation around a vehicle.

  The following vehicle rearward confirmation systems are known. In this vehicle rear confirmation system, an image of the rear of the vehicle taken by a camera is displayed on a monitor so that a driver can visually recognize a person or an obstacle existing behind the vehicle (for example, Patent Document 1).

JP 2004-322799 A

  However, in the conventional system, only the image behind the vehicle is displayed on the monitor, so if the driver cannot see the monitor for a long time while driving, the driver displays the image displayed on the monitor. It was difficult to figure out which position of the eye should be noted.

  The present invention captures an image of the rear of the vehicle to acquire an image, stacks the acquired images of each frame in the time axis direction to generate a solid, and arbitrary pixels and vanishing points of the frame positioned on the surface of the solid An object that includes an arbitrary pixel on a slice image that is cut in the time axis direction along a straight line or curve that passes through, and that is reflected in an arbitrary pixel based on the slope of the straight line and curve that appear in each area Determine whether an object is moving away from the host vehicle, approaching the host vehicle, or moving in parallel with the host vehicle, and apply video processing to the image of each frame. Further, the present invention is characterized in that the attractiveness of a pixel in which an object moving away from the host vehicle is reflected is reduced, and an image after image processing is displayed.

  According to the present invention, since the attractiveness of the pixels in which the object moving away from the host vehicle is reflected is reduced, the attractiveness of the region including the object that is less required by the driver is reduced. Thus, it is possible to prevent the driver from paying attention to an object with a low need for attention. For this reason, the driver can grasp the object to be noted in the video in a short time.

-First embodiment-
FIG. 1 is a block diagram illustrating a configuration of an embodiment of a vehicle monitoring apparatus according to the first embodiment. The vehicle monitoring device 100 includes a rear camera 101, a video processing device 102, and a display device 103. The rear camera 101 is installed at the center of the rear hatch of the vehicle and shoots the rear of the vehicle. The video acquired by the rear camera 101 is output to the video processing device 102.

  The video processing apparatus 102 includes a CPU, a memory, and other peripheral circuits. The video processing apparatus 102 stores video input from the rear camera 101 in a memory (input buffer), and then performs video processing on the video. Then, the video obtained as a result of the video processing is written to the memory (output buffer). Thereafter, the video written in the output buffer is output to the display device 103 at a predetermined frame rate.

  The display device 103 is a liquid crystal display, for example, and is installed at a position such as the center of a dashboard in a vehicle where the driver can visually recognize the driver without moving his / her line of sight while driving. The display device 103 displays the video input from the video processing device 102.

  For example, the rear camera 101 continuously shoots one frame of video as shown in FIG. 2 and outputs the video to the video processing apparatus 102. The video shown in FIG. 2 is a video when the resolution of the rear camera 101 is (xmax, ymax). This image is trimmed by p pixels from the entire periphery by the image processing device 102 described later, and the resolution is converted into (xmax−2 * p, ymax−2 * p) and output to the display device 103.

  In general, in a vehicle that does not include the vehicle monitoring device 100 according to the present embodiment, the driver checks the situation behind the vehicle by looking at the room mirror and the door mirror. However, the installation position and field of view of the room mirror and door mirror in the vehicle are determined by laws and regulations, and the installation position is a position that cannot be visually recognized unless the driver moves the line of sight greatly. For this reason, it is difficult for the driver to grasp in detail the situation behind the vehicle when the line of sight cannot be moved greatly, such as during driving, and there is an object to be noted in the rear of the vehicle. Even if it is, there is a possibility that the object is overlooked.

  On the other hand, when the rear camera 101 captures and displays the rear image of the vehicle on the display device 103 as in the vehicle monitoring device 100 in the present embodiment, the rear image of the vehicle is driven Can be displayed on the display device 103 installed at a position that is easy for a person to see. This makes it possible for the driver to check the situation behind the vehicle without moving the line of sight greatly even during driving, and it is possible to grasp the situation behind the vehicle in detail. It is possible to prevent oversight of things.

  In the vehicle monitoring apparatus 100 according to the present embodiment, the video displayed on the display device 103 is further easy for the user to see, and the user accurately grasps an object that is present behind the vehicle and needs attention. To be able to. For this purpose, the video processing apparatus 102 performs the following video processing on the video captured and input by the rear camera 101.

  FIG. 3 is a block diagram illustrating a configuration of the video processing apparatus 102 according to the first embodiment. Images continuously input from the rear camera 101 are stored in the buffers 1 to n. Specifically, after one frame of video input from the rear camera 101 is stored in the buffer 1, the video is delayed by one frame with a delay. Thereafter, the next input video for one frame is stored in the buffer 2. By repeating this operation, images of n frames, for example, 23 frames are stored in the buffers 1 to n. In the images of the frames stored in the buffers 1 to n, the background and the position of the object change with time as the vehicle travels. Even when the vehicle is stopped, the positions of these objects change with time as other vehicles, pedestrians, and the like move.

  The slice generation unit 102a reads the video for n frames stored in the buffers 1 to n in this way, and generates a slice image to be described later. Specifically, the processing is as follows. First, as illustrated in FIG. 4A, the slice generation unit 102 a stacks a plurality of frames read from the buffers 1 to n in order of acquisition time. That is, a plurality of frames are stacked in the time axis direction by overlapping a frame with a new acquisition time under a frame with an old acquisition time. In the example shown in FIG. 4A, the frame 4a located at the top is the oldest imaged image, and the frame 4b located at the bottom is the latest imaged image. Each frame includes a vanishing point 4c. When a plurality of frames are overlapped, the vanishing point 4c also overlaps at the same position.

  Here, the video processing apparatus 102 forms a solid (cuboid) as shown in FIG. 4B by stacking the video of each frame after adding depth to the video of each frame. In addition, in order to give depth to the video of each frame, each pixel constituting the video of each frame is extended by a predetermined pixel, for example, one pixel in the depth direction. Thus, one pixel in the video can be made a cube having a depth of one pixel, and each frame can be made into a solid body having a depth (thickness) of one pixel as a whole.

  At this time, the video processing apparatus 102 makes sure that the entire cube generated for each pixel in the frame has the color information of the pixel that is the generation source. That is, when the cube generated based on each pixel is viewed from any direction, it has the same color as that pixel in the video, and the cube generated based on each pixel is cut (sliced) at an arbitrary position. In addition, the cut surface also has the same color as the pixel in the video.

  As shown in FIG. 4B, the solid generated by stacking the frames in this way has the horizontal direction of the video as the x axis, the vertical direction as the y axis, and the depth direction as the time axis (t axis). Expressed in three-dimensional coordinates. FIG. 4C shows an example in which the solid shown in FIG. 4B is cut (sliced) in the t-axis direction along a line AB passing through the vanishing point 4c of the video. In this way, when the line AB is cut in the t-axis direction, the cut surface ABCD appears as a two-dimensional image. At this time, as described above, since the entire cube generated based on the pixels constituting each frame has the color information of the pixel that is the generation source, the cut surface image of the cut surface ABCD ( (Slice image) is an image showing temporal changes of pixels existing on the straight line AB.

  FIG. 4D is a diagram illustrating a specific example of an image of the cut surface ABCD. In FIG. 4D, the vertical axis is the x-axis and the horizontal axis is the t-axis. Therefore, the vertical direction of the slice image represents the pixel position existing on the straight line AB in the video of each frame, and the horizontal direction of the slice image represents the pixel existing on the straight line AB between the frames. This represents a change, that is, a time change of pixels existing on the straight line AB. In this slice image, the left of the image is a pixel of the old video, and the pixel of the new video changes to the right.

  In the slice image, as shown in FIG. 4 (d), a plurality of straight lines and curves appear due to the color change of each pixel on the straight line AB between the frames with time. The straight line and the curve indicate a change in position between frames of the background and the object as the vehicle travels and / or a change in position between frames of these objects as the other vehicle and pedestrian move.

  That is, in the image of each frame, an object moving in parallel with the host vehicle, such as an object moving in the same direction or the opposite direction as viewed from the host vehicle, is centered on the vanishing point 4c between frames. Move radially. For this reason, a solid image obtained by stacking a plurality of frames of video is cut in the t-axis direction along a straight line passing through the vanishing point 4c of the video to obtain a slice image, whereby an image of the same part of each object is always displayed on the slice image. The position change can be captured and the position change appears as a straight line or a curve.

  FIG. 5A is an enlarged image obtained by enlarging a part of the slice image shown in FIG. 4D, and FIG. 5B schematically shows a straight line and a curve appearing in the enlarged image. FIG. Here, a straight line and a curve appearing in the enlarged image will be described with reference to FIG. In a magnified image obtained by extracting and enlarging a part of a slice image including a vanishing point 4c of a video when a vehicle equipped with the vehicle monitoring device 100 is traveling on a straight road (when vehicle speed> 0), Since an image of an object moving at the same speed in the same direction as the vehicle does not move between the frames, the line composed of vanishing points 4c of each frame (as indicated by an arrow 5b on the enlarged image) ( The vanishing point line) appears as a straight line parallel to 5a.

  An image of an object moving away from the vehicle, such as an object moving in the same direction as the vehicle at a slower speed than the vehicle, an object moving in the opposite direction of the vehicle, or a stationary object, Since it moves in the direction approaching the vanishing point 4c, it appears as curves 5c, 5d, and 5g approaching the vanishing point line 5a in the enlarged image. On the other hand, an object moving at a higher speed than the vehicle in the same direction as the vehicle, that is, an image of an object approaching the vehicle moves in a direction away from the vanishing point 4c between frames. In the enlarged image, they appear as curves 5e, 5f, and 5h that are away from the vanishing point line 5a.

  Here, although the example which cut | disconnected the solid shown in FIG.4 (b) in the t-axis direction by the straight line AB which passes the vanishing point 4c of an image was demonstrated, as mentioned above, the object which moves in parallel with the own vehicle Moves radially around the vanishing point 4c between frames, so if it is a straight line passing through the vanishing point 4c of the image, even if it is cut in the t-axis direction by a straight line other than the straight line AB Similar straight lines and curves appear on the image and the enlarged image. That is, on the slice image cut in the t-axis direction along the line AB passing through the vanishing point 4c of the video, the image of the object moving in parallel with the host vehicle on the straight road is always a straight line parallel to the vanishing point line 5a. Or a curve approaching the vanishing point line 5a and a curve leaving the vanishing point line 5a.

  Note that, as described above, in the slice image, the video is new as it goes to the right of the image. Therefore, as shown in FIG. 5B, the curve approaching the vanishing point line 5a is the vanishing point line 5a. In the upper region, the curve is a downward-sloping curve, and in the region below the vanishing point line 5a is a upward-sloping curve. Further, the curve away from the vanishing point line 5a is a curve that rises to the right in the region above the vanishing point line 5a and becomes a curve that falls to the right in the region below the vanishing point line 5a.

  As described above, using the slice image obtained by cutting in the t-axis direction along the line AB passing through the vanishing point 4c of the video, the rear camera 101 was photographed based on the slope of the straight line or curve appearing on the slice image. It is possible to identify the movement status of the object included in the video. That is, it identifies whether an object existing behind the host vehicle on a straight road is an object traveling at the same speed as the host vehicle, an object moving away from the host vehicle, or an object approaching the host vehicle. be able to.

  On the other hand, in the case where a slice image is obtained by cutting the solid shown in FIG. 4B in the t-axis direction along a straight line that does not pass through the vanishing point 4c of the image, it moves radially around the vanishing point 4c. Since the image of the same part of the object cannot be captured, the straight line and the curve constituting the image of the same part of the object do not appear on the slice image. That is, it is not possible to identify the movement status of an object existing behind the host vehicle based on the slope of a straight line or a curve that appears on the slice image.

  In the present embodiment, the video processing apparatus 102 identifies the movement status of the object included in the video captured by the rear camera 101 in consideration of the above. Specifically, the processing is as follows. Here, as described above, it is assumed that the vehicle is traveling on a straight road. As described above, the solid is cut in the t-axis direction along the line A-B passing through the vanishing point 4c of the image and sliced. A method for obtaining an image will be described. When a vehicle is traveling on a curved road (curve), the solid is cut in the t-axis direction along a curve corresponding to the curvature of the road passing through the vanishing point 4c of the image. Then, if the slice image is obtained, the moving direction of the object existing behind the vehicle can be similarly determined.

  The slice generation unit 102a calculates the slope of the slice in order to obtain such a slice image. In the present embodiment, for each pixel (pixel) on the plane (video plane) of the topmost video 4d shown in FIG. 4B, a straight line passing through both the pixel and the vanishing point 4c. The slope is calculated as the slope of the slice. If the direction of the rear camera 101 is not directly behind the vehicle or the road is curved, the direction of the optic flow is determined based on the direction of the rear camera 101 and the lens distortion or the curvature of the road. Find and use the slope.

  The slice generation unit 102a sets a neighboring slice region having a calculated slice inclination and a predetermined size. That is, for each pixel (pixel), a slice image is generated by cutting the solid shown in FIG. 4B in the t-axis direction by a straight line passing through both the pixel and the vanishing point 4c. Then, an area of a predetermined size centered on the pixel on the slice image is set as a neighboring slice area. In the present embodiment, the size of the region set as the neighboring slice region is 23 × 23 pixels centering on the target pixel.

  For example, as shown in FIG. 6A, when the target pixel is the pixel 6a, as shown in FIG. 4B, the solid is formed in the t-axis direction by a straight line passing through the pixel 6a and the vanishing point 4c. A cut slice image is generated, and a neighboring slice region 6c including the pixel 6a is set on the slice image. A specific example of the neighboring slice region 6c of the pixel 6a is shown in FIG. As shown in FIG. 6A, when the target pixel is the pixel 6b, as shown in FIG. 6B, the solid is formed in the t-axis direction by a straight line passing through the pixel 6b and the vanishing point 4c. A cut slice image is generated, and a neighboring slice region 6d including the pixel 6b is set on the slice image.

  Next, the filter unit 102b performs video processing on the pixel 6b and the neighboring slice region 6d corresponding to the pixel 6b, and corrects the pixel value (pixel value) of the pixel 6b based on the result of the video processing. Here, as described above, the curve having a slope that approaches the vanishing point line 5a with time on the slice image indicates the movement of the object moving away from the host vehicle, and for a neighboring slice region including such a curve, Thus, video processing is performed so as to suppress high-frequency components with high attractiveness. On the other hand, since a curve having a slope away from the vanishing point line 5a with time on the slice image indicates the movement of an object approaching the host vehicle, the neighboring slice region including such a curve In order to maintain the details necessary for attractiveness and correct recognition, video processing that passes through all high, medium, and low frequencies is applied.

  That is, video processing that reduces the driver's attractiveness is applied to the neighboring slice region including pixels in which an object moving away from the host vehicle is not required to be paid attention. The neighboring slice region including the pixel in which the object that the driver needs to pay attention is subjected to video processing that maintains the driver's attractiveness.

  Therefore, in the present embodiment, the filter unit 102b performs frequency filtering processing using a two-dimensional FIR filter on each neighboring slice region. That is, a two-dimensional frequency filtering process, which will be described later, is applied to a neighboring slice region corresponding to each pixel to perform a two-dimensional frequency filtering process. Then, the pixel value of the pixel 6b after applying the filter is written out at the pixel position in the output buffer. By performing this process on the neighboring slice region set for all the pixels on the image plane 4d, the pixel values of all the pixels constituting the image are corrected.

  Here, the curve having an inclination approaching the vanishing point line 5a with time on the slice image becomes a downward-sloping curve in the region above the vanishing point line 5a as described above with reference to FIG. In the region below 5a, the curve is an upward curve. Further, the curve having an inclination away from the vanishing point line 5a is a curve that rises to the right in the region above the vanishing point line 5a, and is a curve that falls to the right in the region below the vanishing point line 5a.

  For this reason, when the neighboring slice area is included in the area below the vanishing point line 5a, the frequency component of the lower right curve is allowed to pass, and the frequency component of the upper right curve is suppressed almost completely. What is necessary is just to process. That is, among the slopes of straight lines and curves appearing in the neighboring slice area, the pixel values of the pixels constituting the slope of the curve appearing with the movement of the object approaching the own vehicle, and the movement in parallel with the own vehicle The pixel values of the pixels constituting the slope of the straight line appearing with the movement of the target object passing through the pixel values of the pixels constituting the slope of the curve appearing with the movement of the target object moving away from the host vehicle On the other hand, a two-dimensional frequency filtering process may be performed using a filter having a characteristic of performing a low-pass filtering process. Here, a case where a filter having a size of 11 × 11 as shown in FIG. 7 is applied to each neighboring slice region will be described.

  Note that the filter shown in FIG. 7 is a filter having characteristics such that stronger low-pass filtering is performed as the slope of a curve appearing as the object moving away from the host vehicle moves is larger. In the processing, the impulse response of the filter calculated by the Blackman window function is as shown in FIG. This indicates that the input pattern shown in FIG. 8B has a characteristic that the output pattern shown in FIG. 8A can be obtained.

  By comparing FIG. 8A and FIG. 8B, it can be seen that the pattern of the upward curve is suppressed and the pattern of the downward curve passes. That is, by applying this filter to a neighboring slice region that exists below the vanishing point line 5a, the attractiveness of an object that is moving away from the host vehicle is lost, and the region includes only an object that approaches the host vehicle. The attractiveness of can be retained.

  Further, when the neighboring slice region is present above the vanishing point line 5a, video processing is performed so that the frequency component of the upward-sloping curve is passed and the frequency component of the downward-sloping curve is suppressed almost completely. Good. For this reason, by applying a filter having characteristics opposite to the left and right of the filter shown in FIG. 7 to the neighboring slice region located in the region above the vanishing point line 5a, the vehicle is similarly removed from the own vehicle. It is possible to lose the attractiveness of the area including only the moving object and maintain the attractiveness of the object approaching the host vehicle.

  The synthesizing unit 102 c synthesizes each pixel whose pixel value is corrected by the filter unit 102 b into one image and outputs the synthesized image to the display device 103. That is, the pixel value after the video processing is written at the pixel position of the output buffer and synthesized. Then, as described above, the video is converted so as to have a resolution of (xmax−2 * p, ymax−2 * p) and output to the display device 103.

  FIG. 9 is a flowchart showing processing of the vehicle monitoring apparatus 100 according to the present embodiment. The process shown in FIG. 9 is started when the rear camera 101 starts capturing an image behind the vehicle and detects that the number of frames necessary for the process, for example, 23 frames, is stored in the input buffer. Is executed by the video processing apparatus 102.

  In step S10, when 23 frames necessary for processing are stored in the buffers 1 to n (input buffers), the slice generation unit 102a sets the latest frame as the first frame and starts processing the frame. To do. Thereafter, the process proceeds to step S20, the depths of the images of the frames stored in the input buffer are added and stacked to generate a solid as shown in FIG. 4B, and the process proceeds to step S30.

  In step S30, it is determined whether or not processing has been completed for all pixels of the frame being processed. If it is determined that the processing has been completed, the process proceeds to step S31, where the video stored in the output buffer is output to the display device 103, and processing for the next frame, that is, processing for 23 frames including the next frame is performed. To start. On the other hand, if it is determined that the process has not been completed, the process proceeds to step S40.

  In step S40, as described above, for an arbitrary pixel (pixel) on the plane (video plane) of the topmost video 4d shown in FIG. 4B, both the pixel and the vanishing point 4c. A slice image is generated by cutting the solid in the t-axis direction along a straight line passing through. Then, a predetermined area including the pixel on the slice image is set as a neighboring slice area. Thereafter, the process proceeds to step S50, and as described above, video processing is performed on the pixel and a neighboring slice region corresponding to the pixel. Then, the pixel value after the video processing is written to the pixel position of the output buffer. Thereafter, the process proceeds to step S60.

  In step S60, it is determined whether the ignition switch of the vehicle has been turned off. If it is determined that it is not turned off, the process returns to step S20 to repeat the process. On the other hand, if it is determined that it is turned off, the process is terminated.

According to the first embodiment described above, the following operational effects can be obtained.
(1) Since the attractiveness of the area including the object that the driver need not be careful of is reduced, it is possible to prevent the driver's attention from being directed to the object that is not required to be noted. The driver can grasp the target object in the video in a short time.

(2) A plurality of frames shot by the rear camera 101 are stacked in the time axis direction to generate a solid, and the solid is timed by a straight line or curve passing through an arbitrary pixel and vanishing point of the frame located on the surface of the solid. A neighboring slice region including arbitrary pixels was set on the slice image cut in the axial direction. Then, based on the slope of the straight line and the curve appearing in each neighboring slice area, the object shown in the arbitrary pixel is moving away from the host vehicle, approaching the host vehicle, or Judgment is made whether the vehicle is moving in parallel with the vehicle. As a result, when the vehicle is traveling (when vehicle speed> 0), an image of an object moving at the same speed in the same direction as the vehicle is displayed between frames in the slice image including the vanishing point of the image. Since the pixel position does not move, it appears as a straight line parallel to the vanishing point line. Further, since the image of the object moving away from the vehicle moves in the direction approaching the vanishing point between frames, it appears as a curve approaching the vanishing point line in the slice image. On the other hand, the image of the object approaching the vehicle in the same direction as the vehicle moves in a direction away from the vanishing point between frames, so that it appears as a curve away from the vanishing point line in the slice image. In addition, the moving direction of the object can be determined with high accuracy. In addition, by using such a determination method, it is possible to determine the moving direction of an object existing behind the vehicle without installing an expensive sensing device such as a laser radar, thereby reducing the cost of the apparatus. can do.

(3) The region including the object that is less necessary for the driver to pay attention to is an area including only pixels in which the object moving away from the host vehicle is shown. This makes it possible to accurately determine objects that require less attention from the driver, taking into account that objects moving away from the vehicle are generally less likely to pose a danger to the vehicle. Can do.

(4) Of the slopes of the straight lines and curves appearing in the neighboring slice area, the pixel values of the pixels constituting the slope of the curves appearing with the movement of the object approaching the own vehicle, and in parallel with the own vehicle The pixel values of the pixels constituting the slope of the straight line appearing with the movement of the moving object pass through, and the pixel values of the pixels constituting the slope of the curve appearing with the movement of the object moving away from the host vehicle In contrast, two-dimensional frequency filtering processing is performed using a frequency filter having a characteristic of performing low-pass filtering processing. Accordingly, it is possible to reduce only the attractiveness of the region including the object that is away from the host vehicle, which is less necessary for the driver to pay attention to. In addition, the processing load can be reduced because it is possible to reduce only the attractiveness of the region including the object that is less necessary for the driver to pay attention to by simple processing.

(5) When the host vehicle is traveling on a straight road, the solid vehicle is cut in the time axis direction along a straight line passing through an arbitrary pixel in the frame and the vanishing point, and the host vehicle is generated. When traveling along a curve, the slice is generated by cutting the solid in the time axis direction along a curve of curvature corresponding to the curve passing through an arbitrary pixel in the frame and the vanishing point. As a result, a slice image can be generated by an optimum cutting method according to the road that is running. In addition, by using such a slice image, it is possible to accurately determine the moving direction of the object existing behind the host vehicle regardless of whether the host vehicle is traveling on a straight line or traveling on a curve. Can be determined.

-Second embodiment-
In 2nd Embodiment, the process by the filter part 102b mentioned in 1st Embodiment is further simplified, and the attractiveness of the area | region containing the target object with a low necessity for a driver | operator is reduced. In addition, about each figure of FIGS. 1-6 and FIG. 9, since it is the same as that of 1st Embodiment, description is abbreviate | omitted.

  In this embodiment, the slice generation unit 102a sets the above-described neighboring slice region to a size of 5 × 5 pixels with the target pixel as the center. And the filter part 102b correct | amends the pixel value of each pixel in a nearby slice area | region so that the attractiveness of the object which leaves | separates from a vehicle may be reduced. Specifically, the filter unit 102b performs the following process.

  First, as shown in FIG. 10, the filter unit 102b takes four pixels on each of the upper, lower, left, and right sides of the correction target pixel 10a. When the target pixel is located above the vanishing point line 5a, the filter unit 102b calculates the average value of the pixel values of the right four pixels 10b and the lower four pixels 10c as the first value. An average value is calculated, and an average value of the pixel values of the left four pixels 10d and the upper four pixels 10e is calculated as a second average value.

  The filter unit 102b compares each of the first average value and the second average value calculated in this way with the pixel value of the pixel 10a, and out of the first average value and the second average value, the pixel A value closer to the pixel value of 10a is selected as a correction pixel value. Then, a value obtained by multiplying the pixel value for correction and the pixel value of the pixel 10a by a predetermined ratio, respectively, is used as the pixel value of the corrected pixel 10a.

For example, when correction is performed so that the correction pixel value and the pixel value of the pixel 10a have a ratio of 3: 7, the pixel value of the corrected pixel 10a is calculated by the following equation (1).
Pixel value of pixel 10a after correction = pixel value for correction × 0.3 + pixel value of pixel 10a × 0.7 (1)

  In this way, four pixels are provided on the top, bottom, left, and right of the pixel 10a to be corrected, and the average value (first average value) of the right four pixels 10b and the lower four pixels 10c, and the left four The pixel value of the pixel 10a is corrected using the average value (second average value) of the pixel values of the pixel 10d and the upper four pixels 10e. Accordingly, it is possible to correct the pixel value of the pixel located near the boundary between the curve and the background having a slope approaching the vanishing point line 5a with time on the slice image. That is, the outline of the object moving away from the vehicle can be blurred.

  When the target pixel is located below the vanishing point line 5a, the average value of the pixel values of the right four pixels 10b and the upper four pixels 10e is set as the first average value, and the left The average value of the pixel values of the four pixels 10d and the lower four pixels 10c may be calculated as the second average value.

  For example, as shown in FIG. 11, the pixel value of each pixel of the vehicle direction video (input video) captured by the rear camera 101 shown in FIG. Can be recorded in the output buffer. Comparing the input image shown in FIG. 11 (a) with the corrected image shown in FIG. 11 (b), the edge of the vehicle approaching the host vehicle among other vehicles and buildings shown in the image Although it is somewhat blurred, it has not lost its attractiveness, but the buildings in the background that move away from the host vehicle have become more blurred and lose their attractiveness. As a result, the building in the background away from the host vehicle that the driver does not need to pay attention to can make it difficult for the driver's line of sight to be confused, and the driver must pay attention to other vehicles among the objects present in the video. You can figure out what to do.

-Third embodiment-
In the third embodiment, processing when the rear camera 101 has a wide-angle lens and the frame rate of the rear camera 101 is low will be described. In addition, about each figure of FIGS. 1-6 and FIG. 9, since it is the same as that of 1st Embodiment, description is abbreviate | omitted.

  When the camera 101 is equipped with a wide-angle lens, if the frame rate is low, there may be a large shift in the area around the image between successive frames, and the straight line or curve in the slice image may be interrupted There is sex. For example, FIG. 12A shows a composite image in a case where two frames that are continuously shot are superimposed when the frame rate is 60 Hz. In the image shown in FIG. 12 (a), the guardrail poles shown in the peripheral region 12a in the image have moved greatly between the frames, resulting in a large interval. In addition, FIG.12 (b) is the figure which expanded the area | region 12a.

  When the three-dimensional image is generated by superimposing the frames shot at such a low frame rate and the above-described slice image is generated, a slice image as shown in FIG. 13A is obtained. FIG. 13B is an enlarged view of a part of the area above the vanishing point line 5a of the slice image. As shown in FIG. 13B, the curve is interrupted on the slice image.

  FIG. 13C is an enlarged view of a part of the area below the vanishing point line 5a of the slice image, and FIG. 13D is an enlarged view of FIG. 13C. It is the figure which showed the straight line (or curve) which has appeared typically. As shown in FIG. 13D, on the slice image, an original straight line (main pattern) 13a generated as the object moves and a straight line (pseudo pattern) 13b generated in a pseudo manner when the straight line is interrupted. It can be seen that and appear.

  As described above, when the straight line or curve appearing on the slice image is interrupted or the pseudo pattern 13b is generated, the slope of the straight line or curve appearing on the slice image described in the first embodiment is reduced. There is a possibility that an error occurs in the determination of the moving direction of the corresponding object. For this reason, in the present embodiment, the slice generation unit 102a performs a blurring process along the flow of the optic flow on the video of each frame input from the rear camera 101, and then superimposes each frame. Is generated. As a result, the slice image shown in FIG. 13A is corrected as shown in FIG. That is, it is possible to obtain a slice image without the break of the curve shown in FIG. 13B or the pseudo pattern 13b shown in FIG.

  Here, as shown in FIG. 13A, the break of the curve and the generation of the pseudo pattern on the slice image occur in the peripheral area of the video far from the vanishing point line 5a and occur in the vicinity of the vanishing point line 5a. Not done. For this reason, when the blurring process described above is performed on the entire video of each frame input from the rear camera 101, blurring occurs near the vanishing point line 5a as shown in FIG. 13 (e). Resulting in. For this reason, the slice generation unit 102a does not perform the blurring process on the neighboring area 14a of the vanishing point 4c in the video of each frame shown in FIG. 14A, and the peripheral area shown by the oblique lines in FIG. Blur processing is performed on 14b.

  Since the peripheral area 14b set for each frame changes according to the speed of the vehicle, the size of the moving object, and the distance from the vehicle, the slice generation unit 13a previously sets the speed of the vehicle and the size of the moving object. It is preferable to set the peripheral region 14b on the frame after measuring the distance from the vehicle.

  According to the third embodiment described above, the following operational effects can be obtained in addition to the operational effects of the first and second embodiments. That is, when the camera 101 is equipped with a wide-angle lens and the frame rate is low, the surrounding area 14b of each frame is subjected to blurring processing along the optic flow, and then a three-dimensional object is created by superimposing the frames. I tried to do it. For this reason, when the camera 101 is equipped with a wide-angle lens and the frame rate is low, the movement direction of the object is accurately determined by correcting the break of the straight line or the curve generated on the slice image or the generation of the pseudo pattern. Can do.

-Fourth embodiment-
In the above-described first to third embodiments, the filter unit 102b applies the filter shown in FIG. 7 to the neighboring slice area set on the slice image, so that the object moving away from the host vehicle is displayed. The attractiveness is lost, and the attractiveness of the area that contains only the object approaching the vehicle is maintained. On the other hand, in the fourth embodiment, a method for obtaining the same effect without using a filter will be described. 1, 2, 4 to 6, and FIG. 9 are the same as those in the first embodiment, and thus the description thereof is omitted.

  FIG. 15 is a block diagram illustrating a configuration of the video processing apparatus 102 according to the fourth embodiment. In FIG. 4, the same components as those in the block diagram showing the configuration of the video processing apparatus 102 in the first embodiment shown in FIG. The explanation is centered.

  The slice generation unit 102a generates the above-described slice image, and sets a neighboring slice area of a predetermined size, for example, 11 pixels × 11 pixels, with each pixel of the slice image as a center. Then, the set neighboring slice region is output to the inclination detecting unit 102d. The inclination detection unit 102d calculates the inclination of a straight line or a curve included in the target pixel with the central pixel in the neighboring slice area input from the slice generation unit 102a as the target pixel. Hereinafter, processing executed by the inclination detection unit 102d will be described with reference to FIG.

  FIG. 16A is a diagram illustrating a specific example of the neighboring slice region input from the slice generation unit 102a. The inclination detection unit 102d cuts out a plurality of sample areas including the central pixel from this neighboring slice area. For example, as shown in FIG. 16B, first, the inclination detecting unit 102d cuts out a sample region 16b having a height of 11 pixels and a width of 11 pixels including the central pixel 16a of the neighboring slice region in the horizontal direction. Then, the inclination detection unit 102d rotates the sample region 16b by a predetermined angle around the pixel 16a to obtain a plurality of sample regions. In the example shown in FIG. 16B, the sample region 16b is rotated in three stages by 20 degrees clockwise and counterclockwise, respectively, thereby cutting out a total of seven sample regions.

  FIG. 16C shows the sample areas cut out. In FIG. 16C, the sample region 16b is represented as 0, and the sample region obtained by rotating the sample region 16b clockwise by one step (20 degrees) is rotated by +1, two steps (40 degrees). A sample region obtained by rotating +2 to 3 steps (60 degrees) is represented as +3. Further, the sample region obtained by rotating the sample region 16b by one step (−20 degrees) counterclockwise is rotated by −1, two steps (−40 degrees), and the sample region rotated by −2, three steps (−60 degrees). The made sample region is represented as -3.

  Next, the inclination detection unit 102d sets a plurality of shift regions by shifting (shifting) each sample region by a predetermined number of pixels in the horizontal direction. In the present embodiment, each sample area (shift 0) is shifted by 2 pixels to the right (shift 2), shifted by 4 pixels to the right (shift 4), and shifted by 8 pixels to the right. Four shift regions are set: a region (shift 8) and a shift region (shift 16) shifted to the right by 16 pixels. As a result, four shift regions are set for seven sample regions of −3 to +3, respectively. That is, a total of 28 shift areas are set. FIG. 16D shows a specific example of the shift region set by shifting the +3 sample region.

  For each sample area, the inclination detection unit 102d calculates a difference in pixel values between pixels having the same pixel position between the sample area (shift 0) and each shift area. For example, as shown in FIG. 16D, the difference between the pixel values of the pixels having the same pixel position between the +3 sample area (shift 0) and the shift area shifted by 4 pixels (shift 4). Is calculated. This is performed for all the shift regions, and the average value of the pixel value differences calculated as a result is set as a value for determining the inclination of the +3 sample area (inclination determination value). The inclination detection unit 102d calculates the inclination determination value of each sample area of −3 to +3 by performing the calculation of the inclination determination value for all the sample areas.

  Then, the inclination detection unit 102d extracts the sample area having the smallest calculated inclination determination value, and combines the rotation angle of the extracted sample area from the sample area 16b as the inclination value of the straight line or curve included in the target pixel. To the unit 102c. When the inclination determination values of all the sample areas are the same, such as when there is no feature in the video, the inclination detection unit 102d outputs a preset default inclination value to the synthesis unit 102c.

  The synthesizing unit 102c corrects the pixels constituting each frame image input from the buffers 1 to n based on the inclination value input from the inclination detecting unit 102d, and then combines the pixels into one image. To the display device 103. Specifically, by the same method as in the first embodiment, whether the object included in each pixel moves away from the host vehicle based on the tilt value input from the tilt detection unit 102d, It is determined whether the vehicle is approaching or is moving in parallel with the host vehicle.

  For a pixel including an object moving away from the host vehicle, the combining unit 102c has a predetermined number of frames of pixels input before the latest frame in order to reduce the driver's attractiveness. An average value is calculated and used as a corrected pixel value of the pixel. Note that the number of past frames used for calculating the average value of the pixel values is preferably changed according to the inclination value input from the inclination detection unit 102d.

  For example, when the inclination value is large, the object is moving away from the vehicle at a high speed, so the number of past frames to be used is increased to make it more blurred. When the inclination value is small, the object is slowly moving away from the vehicle, so the number of past frames to be used is reduced to reduce the blur amount. Thus, the blur amount can be controlled according to the speed at which the object moves away from the vehicle.

  On the other hand, for pixels that include objects other than objects that are moving away from the host vehicle, the combining unit 102c uses the pixel values of the latest frame as they are without correcting the pixel values. Then, the combining unit 102c combines the corrected pixel values into one image.

  According to the fourth embodiment described above, since it is not necessary to apply a filter for each pixel in order to correct each pixel of the video to be output, the processing can be executed at a higher speed.

-Modification-
The vehicle monitoring apparatus according to the above-described embodiment can be modified as follows.
(1) In the first to fourth embodiments described above, it has been described that the position of the vanishing point in the video is constant for all frames taken by the rear camera 101. However, when pitching occurs in the vehicle due to road undulations, the vanishing point position may be different between frames. Even in such a case, in order to make the vanishing point position in the image constant in all frames, the influence of pitching is eliminated from the image by a mechanism similar to the camera shake correction mounted on a digital camera or the like. The video may be stabilized.

(2) In the first to fourth embodiments described above, the example in which the vehicle monitoring device 100 is mounted on a vehicle has been described. However, the present invention is not limited to this, and the vehicle monitoring apparatus 100 may be mounted on another moving body.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired.

  The correspondence between the constituent elements of the claims and the embodiment will be described. The rear camera 101 corresponds to an imaging unit, the video processing device 102 corresponds to a determination unit and a video processing unit, and the display device 103 corresponds to a display unit. The above description is merely an example, and when interpreting the invention, there is no limitation or restriction on the correspondence between the items described in the above embodiment and the items described in the claims.

It is a block diagram which shows the structure of one Embodiment of the monitoring apparatus for vehicles. It is a figure which shows the specific example of the image | video image | photographed with the rear camera. It is a block diagram which shows the structure of the video processing apparatus 102 in 1st Embodiment. It is a figure which shows the example of a production | generation of a slice image. It is a figure which shows the specific example of the straight line and curve which appear on a slice image. It is a figure which shows the example of a setting of a neighborhood slice area | region. It is a figure which shows the specific example of the filter applied to a neighborhood slice area | region. It is a figure which shows the specific example of the input and output of a video process using the filter shown in FIG. It is a flowchart figure which shows the process of the monitoring apparatus 100 for vehicles. It is a figure which shows the correction method of the pixel value in 2nd Embodiment. It is a figure which shows the specific example of the correction | amendment result of the image | video in 2nd Embodiment. It is a figure which shows specifically the problem which generate | occur | produces when the camera 101 mounts a wide angle lens and a frame rate is low. It is a figure which shows the specific example of the slice image in 3rd Embodiment. It is the figure which showed concretely the peripheral region 14b made into the object of the blurring process in 3rd Embodiment. It is a block diagram which shows the structure of the video processing apparatus 102 in 4th Embodiment. It is a figure which shows the specific example of the process performed by the inclination detection part 102d in 4th Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Vehicle monitoring apparatus 101 Rear camera 102 Image processing apparatus 102a Slice production | generation part 102b Filter part 102c Composition part 102d Inclination detection part 103 Display apparatus

Claims (6)

Photographing means for capturing an image of the rear of the vehicle and acquiring images;
The images of the frames acquired by the photographing means are stacked in the time axis direction to generate a solid, and the solid is a straight line or curve passing through an arbitrary pixel and vanishing point of the frame located on the surface of the solid. An area including the arbitrary pixel is set on the slice image cut into two, and the object reflected in the arbitrary pixel moves away from the own vehicle based on the slope of the straight line and the curve appearing in each area. A determination means for determining whether the object is approaching the host vehicle, or is moving in parallel with the host vehicle;
Video processing means for performing video processing on the video of each frame to reduce the attractiveness of pixels in which an object moving away from the host vehicle is shown;
A vehicle monitoring apparatus comprising: display means for displaying an image after image processing is performed by the image processing means. The vehicle monitoring device according to claim 1,
The image processing means performs a two-dimensional frequency filtering process on the region set on the slice image, thereby attracting a pixel in which an object having a low need for attention by the driver is shown. The vehicle monitoring apparatus characterized by lowering. The vehicle monitoring device according to claim 2,
The image processing means is a pixel of pixels constituting a slope of a curve appearing with movement of an object approaching the host vehicle out of slopes of a straight line and a curve appearing in an area set on the slice image The pixel values of the pixels constituting the value and the slope of the straight line appearing with the movement of the object moving in parallel with the host vehicle pass through, and with the movement of the object moving away from the host vehicle A vehicular monitoring apparatus, wherein the two-dimensional frequency filtering process is performed using a frequency filter having a characteristic of performing a low-pass filtering process on a pixel value of a pixel constituting a slope of an appearing curve. The vehicle monitoring device according to claim 3,
The vehicle monitoring device according to claim 1, wherein the frequency filter is a filter having a characteristic such that a stronger low-pass filtering is performed as a slope of a curve appearing as the object moves away from the host vehicle is larger. In the vehicle monitoring device according to any one of claims 1 to 4,
The vehicular monitoring device, wherein the video processing means performs a blurring process along the flow of an optical flow on the video of each frame acquired by the photographing means. Take a picture of the back of the vehicle
A slice obtained by stacking the acquired images of each frame in the time axis direction to generate a solid, and cutting the solid in the time axis direction by a straight line or a curve passing through any pixel and vanishing point of the frame located on the surface of the solid An area including the arbitrary pixel is set on the image, and based on the slope of the straight line and the curve appearing in each area, whether the object shown in the arbitrary pixel moves away from the own vehicle or not. Determine whether it is approaching the vehicle or moving in parallel with the host vehicle,
Video processing is performed on the video of each frame to reduce the attractiveness of pixels in which an object moving away from the host vehicle is shown,
A monitoring method for a vehicle, comprising displaying an image after image processing.

Images