KR101117969B1 - Vehicle surrounding information service system - Google Patents

Abstract

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a road surface information providing apparatus around a vehicle, wherein after obtaining an overhead view image from an image photographed by an imaging unit, an overhead view image is analyzed to detect a road surface state. A road surface information providing apparatus around a vehicle for detecting and providing an area having a lower height than a portion thereof. The vehicle surrounding road surface information providing apparatus of the present invention includes: imaging means for photographing a vehicle periphery; Vehicle information acquiring means for acquiring vehicle state information; Process the image photographed by the imaging means to generate an overhead view, extract a feature point from the generated overhead view, and then make a road surface around the vehicle based on the movement of the feature point and the vehicle state information acquired from the vehicle information acquisition means. An electronic processing apparatus for determining the three-dimensional object or the groove portion present on the image and acquiring and providing the result as road surface information; And notifying means for outputting road surface information acquired and provided by the electronic processing apparatus and providing the same to a driver or a passenger.

Road surface information, bird's-eye view, top view, image pickup means, electronic processing apparatus, image processing unit, groove part, pool, feature point, search, frame

Description

Vehicle surrounding information service system {Vehicle surrounding information service system}

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for providing road surface information around a vehicle, and more particularly, to an apparatus for providing road surface information around a vehicle on the basis of an image obtained from an image pickup means for photographing the vehicle surroundings.

Recently, due to the development of image processing technology and arithmetic technology for photographing and processing an image, an automobile has an image display device for displaying a top view of a vehicle and its surroundings as a top view. It is developed and used.

The image display device synthesizes an image captured by a plurality of cameras, i.e., a plurality of cameras, and a substitute image of the vehicle, and displays an overhead view of a form in which the vehicle is viewed from a virtual point of view over the vehicle. It is a system provided to the driver.

Such a system may include a system that detects an obstacle around a vehicle and provides a driver with an overhead view showing the image, and such a system typically includes an imaging unit, an obstacle detection unit, an electronic processing device for generating an overhead view, and a display unit ( display) as the main configuration.

The electronic processing apparatus includes a memory in which a substitute image for image processing is stored, an image captured by the image pickup means, is converted into an overhead view image, and the replacement image of the vehicle (the image representing the vehicle in the overhead view image) and the obstacle image And an image processing unit for generating a composite image of the image (which is displayed to be recognized as an obstacle in the overhead view image).

When the image processing unit processes the image photographed by the imaging unit and recognizes an obstacle around the vehicle through the obstacle detecting unit, the image of the obstacle is added to the overhead image obtained from the image captured by the imaging unit together with the substitute image of the vehicle. By synthesizing together, a final overhead image is displayed together with the vehicle and the obstacles around it, and then provided to the driver through the display means.

The display means is an in-vehicle monitor, which receives and displays an image generated by an image processor.

On the other hand, the conventional surrounding image display apparatus around the vehicle displays the three-dimensional obstacles and three-dimensional objects protruding from the road surface in the image or provide information such as the road width as an image, preventing the vehicle from contacting the obstacle while driving the driver To assist with manipulation.

In addition, in the conventional image display apparatus around the vehicle, the development is focused on displaying the obstacle more accurately in displaying the obstacle, displaying the obstacle in the non-imageable part, or displaying the obstacle by distinguishing the type of the obstacle. come.

For example, as a prior art, Patent Document 1 discloses an image using an edge position on the vehicle side of the detected obstacle when the obstacle is detected with respect to an area of the obstacle (backside of the obstacle as viewed from the camera direction) where the camera cannot capture. Techniques for specifying a deficiency area and filling it with a replacement image are described.

Patent Literature 2 describes a technique in which a typical picture display (static substitute image) is displayed at a detected position according to the type of obstacle (human, bicycle, vehicle, etc.) detected when an obstacle is detected.

Patent Literature 3 also describes a technique for showing the moving situation as a substitute image on the top view when the detected obstacle (person) is outside the range of the image displayed.

However, as described above, the conventional image display apparatuses mostly have a display function for protruding three-dimensional obstacles around a vehicle, and detect a road surface condition such as a puddle existing on the road surface or a recessed part of the road surface. This cannot be provided to the driver through an overhead view of the top view.

(Prior art technical literature)

(Patent literature)

(Patent Document 1) Japanese Unexamined Patent Publication No. 2003-189293

(Patent Document 2) Japanese Unexamined Patent Publication No. 7-223488

(Patent Document 3) Japanese Unexamined Patent Publication No. 2007-295043

Accordingly, the present invention has been made in consideration of the above-mentioned point, and by detecting grooves on a road or pit in a periphery of a vehicle and displaying them on a top-view overhead view instead of a separate side view image. Another object of the present invention is to provide a vehicle surrounding information providing device for providing road surface information so as to be cautious in driving a driver or getting off a passenger.

In order to achieve the above object, the present invention includes an image pickup means for photographing the surroundings of the vehicle; Vehicle information acquiring means for acquiring vehicle state information; Process the image photographed by the imaging means to generate an overhead view, extract a feature point from the generated overhead view, and then make a road surface around the vehicle based on the movement of the feature point and the vehicle state information acquired from the vehicle information acquisition means. An electronic processing apparatus for determining the three-dimensional object or the groove portion present on the image and acquiring and providing the result as road surface information; It provides a road surface information providing apparatus around the vehicle comprising a notification means for outputting the road surface information obtained and provided by the electronic processing device to provide to the driver or passenger.

Here, the electronic processing apparatus searches for the position of the feature point extracted from the overhead view image one frame before, in the overhead view image of the current frame, and the movement direction of the feature point and the moving direction of the vehicle, the displacement amount or the movement of the feature point one frame before and in the current frame. The speed, the amount of movement of the vehicle or the speed of the movement are analyzed to determine the three-dimensional object or groove on the road surface around the vehicle.

In addition, when the electronic processing apparatus determines the groove portion and determines that the front wheel or the rear wheel of the vehicle stops while passing through the groove portion, the suspension stroke amount measured by the vehicle information acquisition means when the front wheel or the rear wheel passes. In addition to the depth of the groove portion obtained from the change of can be provided to guide the driver or passenger via the notification means according to the groove portion and the stop position.

Here, the electronic processing apparatus may guide the driver to adjust the vehicle stop position when notifying through the notification means.

In addition, the electronic processing apparatus may determine whether the feature portion is a puddle by detecting the luminance according to the reflection of external light from the feature assembly and the road surface portion in the desensitized image in the state where it is determined that the actual road portion corresponding to the feature point is equal to the road surface. Can be.

In addition, if the electronic processing apparatus determines the feature point portion as a pool and determines that the front wheel or the rear wheel of the vehicle stops after passing through the pool, the suspension stroke measured by the vehicle information acquisition means when the front wheel or the rear wheel passes. In addition to the depth of the puddle obtained from the volume change can be provided to inform the driver or passenger through the notification means according to the puddle and the stop position.

Here, the electronic processing apparatus may guide the driver to adjust the vehicle stop position when notifying through the notification means.

Accordingly, the surrounding image display apparatus of the present invention extracts and analyzes feature points from an overhead view image obtained from an image captured by the imaging means, detects three-dimensional objects, grooves, pools, etc. existing on the road around the vehicle therefrom, By providing the detected road surface information through the notification means, the driver can check the road surface surrounding the vehicle and at the same time use it in driving operation.

Particularly, in the present invention, after analyzing the movement of a feature point in an overhead view, detecting a groove or a puddle, information such as shape or position can be provided to the driver or passenger through a notification unit such as a display unit through the overhead view of the top view. This allows the driver to check the road condition that is the obstacle of driving and to operate the driver to avoid it.

In addition, it provides road surface information such as grooves and puddles around the vehicle, and calls attention to help drivers or passengers get off the grooves and puddles safely when the vehicle is stopped. It is advised to adjust the bar, it can contribute to relieve discomfort and safety of the driver or passengers when getting off.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for providing road surface information around a vehicle, and relates to an apparatus for acquiring and providing road surface information around a vehicle based on an image obtained from an image pickup means for photographing the vehicle surroundings.

Furthermore, the present invention obtains an overhead view image from an image photographed by the imaging means, and then analyzes the overhead view image to detect a road surface state, in particular an area having a lower height than other parts of the road surface, such as a recessed area on the road surface around the vehicle. It is configured to detect and provide it to the driver.

In the present invention, in providing road information to a driver, a recessed area may be combined with an overhead view and output through a display (in-vehicle monitor), and visually provided to the driver or passenger for attention. Other notification means, such as speakers, buzzers, lamps, etc., may be used to provide road surface information.

It is difficult to grasp and provide the height information of a display object in the conventional system which provides a top-down view image. In particular, it is not possible to detect road surface areas (grooves, such as areas recessed in the road surface) which are obstacles to running through the captured image information, and necessary road surface information cannot be displayed on the overhead view image.

In order to solve the above problems, in the present invention, an overhead view image is obtained from an image photographed by an imaging unit, and then analyzed, and a three-dimensional object (obstacle / projection), a groove, a puddle, etc. are identified on a road surface, and a groove, a puddle, and the like are driven. It is configured to detect the shape of the road surface which is an obstacle of, and to determine the depth of the groove portion or the pool and provide the result as road surface information.

Accordingly, the driver or the passenger may be careful when driving or getting off with reference to the road information provided. For example, when the groove or the pit around the vehicle is identified from the provided road information, the driver can perform a driving operation to avoid this, and when the driver gets off, the driver and the passenger are notified of the road surface around the vehicle and the pit or the puddle You can get off safely.

Hereinafter, in the present specification, for the sake of clarity, a recessed area having a height lower than that of a flat main surface, that is, having a depth, is called a groove, and a water-filled part such as rainwater is divided into a recessed area to distinguish the groove from a puddle (described below). As detected by the same height as the road surface) will be described.

First, FIG. 1 is a block diagram showing the configuration of a vehicle surrounding road surface information providing apparatus of the present invention.

As shown, the vehicle surrounding road surface information providing apparatus according to the present invention comprises an image pickup means 10 for taking a picture of a vehicle periphery, and an electronic device for processing roadside image information obtained by the image pickup means 10 to obtain road surface information around the vehicle. Acquiring status information such as the processing device 20, the notification means 30 outputting the road surface information acquired and provided by the electronic processing device 20, and providing to the crew such as a driver or a passenger, the moving direction and the moving speed of the vehicle. It may be configured to include a vehicle information acquisition means 40 for.

Here, the imaging means 10 may be a plurality of cameras 11 to 14 installed in the vehicle as means for acquiring an image around the vehicle, and transmits the acquired image data to the electronic processing apparatus 20. Enter it.

The electronic processing apparatus 20 may be configured to process the image data acquired by the imaging means 10 to generate an overhead view on which road surface information is displayed, and the notification means 30 is the electronic processing apparatus 20. May include a display 31 for generating and outputting an overhead view on which road surface information is displayed.

Here, the electronic processing apparatus 20 converts and generates a memory (not shown) in which a replacement image for image processing is stored, and an image taken by the imaging means 10 into an overhead view image, and analyzes the generated overhead view image. An image processing unit 21 which detects obstacles (stereos / projections), grooves or pits on the road surface and displays road surface information such as a replacement image of the vehicle and detected obstacles, grooves and pits on the overhead view image. It is configured to include).

The memory stores a vehicle replacement image (an image representing the vehicle in the overhead view image) for representing the own vehicle in the generated overhead view image, and stores various data and programs necessary for driving the apparatus.

The image processing unit 21 processes the image photographed by the imaging unit 10 and at the same time acquires road surface state information (obstacles, grooves, puddles, etc. information), which is obtained from the image photographed by the imaging unit 10. The roadside information is displayed at the same time as the replacement image of the vehicle is synthesized on the overhead view image, and a final overhead view image in which the vehicle and the road surface information around it are generated is generated, and then provided to the driver through the display unit 31.

At this time, as will be described later, the image processing unit 21 generates an overhead view image from the image acquired through the imaging unit 10 and extracts feature points in the overhead view image, and analyzes them, but analyzes the state of the feature points of one frame before and the current frame. The situation around the vehicle such as the road surface state is determined. In addition, the situation around the vehicle is determined using the vehicle state information input from the vehicle information acquisition means 40 together with the image data acquired by the imaging means 10.

The display unit 31 is an in-vehicle monitor, which receives and outputs a final overhead view image generated by the image processing unit and displaying road surface information.

In addition to the display unit 31, a speaker 32, a buzzer 33, a lamp, or the like may be used as a separate means for transmitting a situation around the vehicle to a driver or a passenger. All available information providing means such as a display unit 31, such as an in-vehicle monitor, which outputs an image and providing information, a speaker 32, a buzzer 33, and lamps, which output information by outputting a sound such as a voice. This will be referred to collectively as the notification means (30).

The vehicle information acquisition means 40 includes a vehicle speed detection unit 41 for detecting a vehicle speed, a steering angle detection unit 42 for detecting a steering angle, a shift position detection unit 43, a height sensor 44, It may be a door lock switch (45).

Hereinafter, a process of detecting and processing a road surface state in the present invention will be described in more detail.

The image processing unit receives the image captured by the camera, converts the image into a bird's-eye view, and synthesizes a substitute image of the vehicle on the converted bird's-eye view to generate a bird's-eye view showing the vehicle and its surroundings. It is the same as the process of generating the overhead view image of i).

However, in the present invention, the principle of determining the property of the object from the feature point in the overhead view image is used (basic principle used: feature point extraction principle by a corner-detector).

That is, the image processing unit detects a feature point from each frame of the overhead view image obtained from the image photographed by the camera and analyzes it to detect obstacles (stereoscopic objects / protrusions) or grooves around the vehicle on the road surface and display them on the overhead view image. It is.

At this time, the road surface state of the road is determined based on the movement of the feature points extracted from successive frames of the overhead view image. When analyzing the movement of the feature points in the successive frames of the overhead view image, the height is lowered compared to the surrounding road surface. It is possible to discriminate road conditions such as three-dimensional objects (including protrusions on the road surface or overall three-dimensional obstacles such as humans and other vehicles) that are higher than grooves and surrounding road surfaces.

In this way, in order to determine the road surface state, the image processing unit can determine that the feature point is high or low with respect to the road surface by obtaining the displacement amount or the moving speed from the position one frame before the feature point extracted from the overhead view image.

2A is a view illustrating the movement of feature points with respect to a three-dimensional object and a road surface in the overhead view image, and FIG. 2B is a view illustrating the extraction of feature points and the detection of feature points in FIG. 2A separately. The operation (on the display) of the display) is shown differently. 2a and 2b show the movement of the feature point on the three-dimensional object and the road surface, in Figs. 2a and 2b, ① represents a feature point extraction and ② represents a feature point search (search). In addition, A represents the feature point of a three-dimensional object, B represents the feature point on the road surface.

3 is an exemplary view showing a state in which a still object is seen according to a change in the position of a viewpoint, and FIG. 4 is a diagram illustrating an example of determining a road surface state around a vehicle from the movement of a feature point.

Referring to FIGS. 2A, 2B, 3, and 4, the principle of determining whether a property of an object, that is, a three-dimensional object higher than the surrounding road surface or a groove portion lower than the surrounding road surface is explained from the movement of the feature point in the overhead view image. .

According to the present invention, the road surface shape and state around a vehicle such as a three-dimensional object and a groove part on a road are classified based on the movement of a feature point in an overhead view image. First, an image processing unit receives images captured by a plurality of cameras and converts the images into an overhead view image. The feature point is recognized and extracted from the converted overhead view image.

Then, the image processing unit searches for the feature points in the overhead view, and searches for the movement of the feature points extracted in ① of FIGS. 2A and 2B, but searches for a position where the feature points extracted in ① are moved after one frame.

As a result, as shown in ②, once the moving position after one frame of the feature point extracted in ① is retrieved, the image processing unit is located between the first position of the feature point (position of ①) and the position of the feature point retrieved after one frame (position ②). The displacement amount and the direction of movement are determined to determine the shape and state of the actual road part corresponding to the feature point.

In the overhead view of the top view, the image of the camera is calibrated according to the road surface, and the visible shape of the object varies depending on the height and position of the object.

Referring to FIG. 3, if it is lower than the installation position of the camera, it is closer to the camera because the height is higher with respect to the road surface at the same position, so that the overhead view image accompanying the movement of the camera position (the movement distance of the camera and the movement distance of the vehicle is the same). The displacement amount of the three-dimensional object (the displacement amount of the feature points A and A 'in Figs. 2A and 2B) is larger than the displacement amount (the displacement amount of the feature points B and B') on the road surface (the surrounding flat road surface rather than the solid body and the groove part), The larger the displacement amount, the higher the height of the three-dimensional object. Similarly, in the case of the groove portion having a low height with respect to the road surface, the displacement amount in the overhead view image is larger than the displacement amount of the position on the road surface, and the lower the height (the deeper depth).

However, in the overhead view image, as shown in Fig. 4, a thing higher than the road surface (stereoscopic) appears to move in the opposite direction to the moving direction of the camera (the moving direction of the vehicle), and the lower one (groove) moves in the same direction. Seems to be.

Referring to FIG. 4, the properties of the feature points can be classified into the following three types by calculating the movement amount (displacement amount) and the direction of movement of the feature points based on the positions of the feature points one frame before and the current feature points.

-Points higher than the road surface: overall obstacles (including human back motion)

-Points on the road: road signs, manholes, road joints, pools filled with water, etc.

-Lower point than road surface: Groove, recessed terminal, etc.

In the case of the point higher than the road surface, the position change is larger than the position on the road surface (normal flat road surface), the moving direction is opposite to the vehicle, and the longer the search distance (the moving distance of the feature point, that is, the displacement amount) is higher from the road surface. .

On the other hand, in the case of the point lower than the road surface, the position change is similarly larger than the position on the road surface, but the moving direction is in the same direction as the vehicle, and the longer the search distance, the lower the road surface.

The method of calculating the actual height from the moving distance of a point in an image is described in the patent document (Japanese Patent Laid-Open No. 2008-99136), and the method of obtaining the depth of a specific position in the groove portion is as follows.

5 is a view for explaining a method of calculating the depth of a portion corresponding to the feature point from the movement of the feature point representing the groove in the overhead view image, wherein each reference numeral shown in the figure is as follows.

B: Point of groove injury

D: travel distance of vehicle

L: Distance of the camera and point B at time T1

B ': position of the point B on the display image (reduced image) at time T1

B ": Position of point B on the display at time T2

L1: distance B 'from the camera at time T1

L2: distance of B "from camera at time T2

H: Depth of point B on the road surface (height to road surface)

h: camera height

Here, B 'may correspond to the position of the feature point extracted one frame before, and B' may correspond to the position of the feature point moved in the current frame. L1 to L2 represent the amount of displacement (displacement) of feature points of one frame and the current frame.

In this case, L1 and L2 in FIG. 5 may be expressed as in the following equations.

Further, from the above equations (1) and (2), the position change amounts L1-L2 on the top view display image (reduction of view images) can be expressed by the following equation.

On the display, it can be seen from Equation 3 that the movement amount (displacement) L1-L2 of the point existing at the position lower than the road surface is smaller than the movement amount (movement distance) D of the vehicle, and the deeper the depth H of the point at the lower position is displayed. The amount of shift of the phase becomes smaller. Therefore, when the movement amount (displacement amount) of the feature point on the display is smaller than the movement amount (movement distance) of the vehicle, it is possible to determine that the point on the actual road surface corresponding to the feature point is located in the groove portion, and thus, as the road surface state around the vehicle, Detection becomes possible.

Depth H from the equation (3) can be represented by the following equation (4).

Depth H can be calculated from Equation 4, and when the moving distance D of the vehicle and the movement amount (displacement) L1-L2 of the feature point are obtained, the depth H can be calculated from the images of the two frames.

The moving distance D of the vehicle is a value that can be calculated while driving, and the camera height h is a fixed value known as the height at which the camera is installed in the vehicle.

As a result, the height of the feature point with respect to the road surface can be determined from the displacement amount of the feature point extracted from the image before and after one frame and the vehicle moving distance at this time, and the road surface surrounding the vehicle obtained through the display unit (monitor), speaker, etc. Through the means of notification of the driver and passengers can be informed.

6 is a diagram illustrating an example in which the determination result of the road surface state is reflected and displayed on an overhead view image.

On the other hand, if the detected feature point is determined at a low position with respect to the road surface, the shape of the pond can be specifically determined based on the luminance information of the image. Since the pond is an area filled with water, when the feature point is analyzed to determine the high and low, the feature point analysis is performed on the surface of the water, so that the puddle is detected at the same height as the surrounding road surface. Therefore, in the case of a pond, it can discriminate | determine by grasping a change of brightness | luminance as mentioned later.

Obtaining the shape of the puddle can inform the driver of the condition and call attention to the road surface, which is an obstacle to driving, and can call attention when the driver or passenger gets off after the vehicle stops.

A process of detecting a puddle existing on a road surface is described below. In this process, it is determined what feature points detected on the screen exist on the road surface, and at this time, the feature points on the road surface and the luminance change of the surrounding road surface are determined.

7 is an example showing a change in luminance when reflecting external light (sunlight, etc.) from a puddle, a manhole, and a road surface. When a puddle or a manhole exists on the road surface, a portion corresponding to the puddle or a manhole in the image obtained within the imaging range of the camera has a change in luminance. On the other hand, in the case of the road surface, there is almost a constant tendency without a change in the luminance due to the external light reflection.

Using this point, the luminance ratio between the feature point and the surrounding flat road surface is calculated from the plurality of image data acquired by the camera in the process of moving the camera position by vehicle movement, and the pool, manhole, and the flat road surface are calculated from the calculated luminance ratio. Determine.

Since the luminance ratio of the pond is relatively larger than that of the manhole, a setting value for distinguishing the manhole and the pool is provided so that the portion where the luminance ratio is higher than the set value can be determined as the pool.

That is, a feature point on a road surface is extracted from an overhead view obtained from an image captured by a camera, a luminance ratio between the road surface and a feature point is calculated, and the type of the feature point is determined according to the luminance ratio. In the puddle, the portion where the luminance is changed but the luminance ratio is lower than the set value is determined by the manhole.

When the pond is discriminated in this way, it becomes possible to display the reflection of the shape of the pool in the overhead view image. As a result, when getting off, it is possible to inform the driver or passenger of the road surface around the vehicle, i.e., there is a puddle around the vehicle. In this case, a speaker, a buzzer, a lamp, or the like can be used as a notification means.

The size and depth of the puddles and grooves present on the road surface can be detected at the stop position. In particular, it is possible to call attention by notifying the driver or passenger who gets off and informs the road surface information around the vehicle.

8 is a diagram illustrating an example of detecting a road surface state based on image processing and vehicle behavior information. After analyzing a feature point of a road surface and determining that a specific point of the road surface is a puddle or a groove, the vehicle passes through the point. When measuring the depth from the change in the suspension stroke amount and when the vehicle stops near the pool or the groove, if necessary, the driver or the driver can be informed through the in-vehicle notification means such as a display unit, a speaker, a buzzer, and a lamp. It may be configured to alert the passengers.

Since the depth of the detected puddles or grooves cannot be accurately known through image processing alone, the electronic processing device (image processor) changes the stroke amount of the suspension through the vehicle information acquisition means (height sensor) when the vehicle passes through the puddles or grooves. Measure the depth of the puddle or groove from thereafter, and notify the driver of the presence, size and depth of the puddle or groove in the state where the vehicle has stopped after passing through the puddle or groove. You will be informed.

Here, the image processing unit may notify the driver to adjust the stop position through a monitor or a speaker, a buzzer, a lamp, etc., when the vehicle stops near a feature point with a large change in stroke amount. When the lock is released, the driver or passenger may be warned to pay attention to the foot when getting off through the display unit (monitor), the speaker, the buzzer, or the lamp closest to the corresponding door. At this time, whether or not the feature is a puddle or groove, and if it is a puddle or groove, the depth and the degree of size can be notified by the size or type (content) of voice or sound.

In this way, when the pool or the groove is present on the road surface at the disembarkation position, attention can be reliably brought to the driver or passengers by emphasizing and informing the condition. In addition, if the size and depth of the pit or the groove portion is less than the set value which does not matter when getting off in a state where it is determined that the pit or the groove portion is present at the disembarkation position, it is also possible to avoid the hassle by notifying the state.

9 is a diagram illustrating an example of detecting surrounding obstacles and calling attention as necessary. In a place where a stop space is limited, such as a parking lot, the driver may be limited in adjusting the vehicle position even if a puddle exists around the vehicle. In this situation, if the vehicle position is excessively adjusted in such a situation, a collision with surrounding obstacles may occur. Therefore, in this case, the notification for adjusting the stop position can be omitted.

In this process, it is determined through the aforementioned feature point extraction and analysis process that the three-dimensional objects (overall obstacles including vehicles or people) and the puddles or grooves are detected around the vehicle, and the distance between the three-dimensional objects before and after the vehicle is paused. After the measurement, if the pit or the groove portion in the disembarkation position is present on the road surface, the notification to adjust the stop position is omitted, and as shown in the description of FIG.

In this way, when both the obstacle and the groove (or the pool) are detected, it is determined whether the notification government is necessary from the positional relationship between the obstacle and the vehicle, and then the unnecessary operation of the notification means is restricted. When the vehicle is stopped in an area with a small margin of adjustment of the stopping position, there is little need to transmit the stopping position adjustment to the driver, thereby reducing the inconvenience of transmitting unnecessary information of the stopping position adjustment.

On the other hand, Figure 10 is a flowchart showing a process of determining the high and low of the actual road portion corresponding to the feature point through the feature point extraction and analysis, as described above, the image around the vehicle is obtained by the image pickup means and the image When input to the processing unit, the image processing unit generates an overhead view image from the image around the photographed vehicle, and then extracts the feature point.

Then, the image processing unit searches the position of the feature point extracted from the previous frame in two consecutive frames in the current frame, determines the moving direction of the vehicle from the vehicle information acquiring means, and then the moving direction of the vehicle is the feature point one frame before on the overhead view image. It is determined whether the direction is the same as the direction from the position of to the position of the current feature point.

If they do not coincide, i.e., in opposite directions, it is determined that the actual part of the road corresponding to the feature point is higher than the surrounding road surface. On the other hand, if they match, that is, in the same direction, the image processing unit calculates the displacement amount of each feature point and the movement amount of the vehicle in the before and after frames. Comparing the displacement amount and the movement amount, if it is equal to each other, it is determined that the actual road portion corresponding to the feature point is at the same height as the surrounding road surface, and if different (displacement amount is smaller than the movement amount), the actual road portion corresponding to the feature point is the surrounding road surface. The lower part is determined. If the actual road surface part is given, the part higher than the road surface becomes a three-dimensional object (obstacle), and the lower part becomes a groove part. Here, after calculating the moving speed of the feature point, that is, the moving speed and the moving speed of the vehicle in the image, it is also possible to compare the two speeds and determine the same height if they are the same and the part lower than the road surface if the difference is different. The characteristic speed of the lower part of the road is slower than the speed of the vehicle.

Next, FIG. 11 is a flowchart illustrating a process of determining whether or not a feature point is a puddle using luminance according to reflection of external light, which is a process proceeding after ① in FIG. 10.

The image processing unit judges that the actual road portion corresponding to the feature point is the same height as the road surface, and then measures the size of the aggregation portion of the feature point. At this time, if the size of the collection portion of the feature points is smaller than the predetermined value (for example, the tire installation area), the display is not displayed. If the size is larger than the predetermined value, the next step is performed.

Then, the image processing unit measures the average luminance of the corresponding area of the road surface of the image data, measures the average luminance of the aggregation portion of the feature points, and then calculates the luminance ratio between the road surface portion and the feature point portion.

When the luminance ratio is smaller than the set value, it is determined that the feature point portion is a road sign, a manhole, or the like. On the other hand, when the luminance ratio is greater than or equal to the set value, it is determined that the feature point portion is a pool, and the pool detection area is highlighted and displayed on the overhead view image.

Next, the attached FIG. 12 accurately measures the depth from the road surface based on the change in the stroke amount of the suspension when detecting a feature point of a height equal to or lower than the road surface (a flat road surface around the road). It shows the process of calling attention through.

When the image processing unit detects the presence of the pool or the groove in the vehicle traveling direction, it displays the detection result on the display and determines that the front wheel (or the rear wheel) has passed through the pool.

Subsequently, the depth of the pool or groove is measured from the change in the amount of suspension stroke when passing through the feature point detected by the height sensor of the vehicle information acquisition means, and then the rear wheel (or the front wheel) after the front wheel (or the rear wheel) passes through the pool or the groove. It is determined that is stopped before passing the feature point.

When the vehicle is stopped, the notification means sets the notification contents, the volume, etc. according to the shape, depth, stop position, etc. of the pits or grooves of the feature points, and notifies the driver when the stop position adjustment is necessary.

If the vehicle information acquisition means detects that the door is no longer moved and the door is unlocked, use the notification means closest to the door (monitor, speaker, buzzer, lamp, etc.) Give notice to be careful.

Next, FIG. 13 shows whether or not the notification means is activated or deactivated depending on whether the vehicle stop position adjustment is possible when the obstacles and the pools (or grooves) are detected together around the vehicle, for example, in parallel parking or parallel parking of the vehicle. An example is shown.

When the obstacle (stereoscopic) and the pool (or the groove) are detected at the same time around the vehicle, and then the vehicle is detected by the vehicle information acquisition means, the image processing unit measures the distance between the vehicle and the surrounding obstacle, and then It is determined whether the stop position of the vehicle can be adjusted so as to avoid a puddle or a groove part when getting off.

If it is determined that the vehicle stop position can be adjusted, the driver is alerted by notifying the driver to adjust the vehicle stop position through a notification means (monitor or speaker, etc.).

On the other hand, if it is determined that the pool cannot be avoided even if the vehicle stop position cannot be adjusted or if the vehicle stop position is slightly adjusted within a range where obstacles can be avoided, no attention is given to the driver to adjust the vehicle stop position.

1 is a block diagram showing the configuration of a vehicle surrounding road surface information providing apparatus of the present invention;

2A is a diagram illustrating the movement of feature points on a three-dimensional object and a road surface in an overhead view image;

FIG. 2B is a diagram illustrating the feature point extraction and the feature point detection of FIG. 2A separately; FIG.

3 is an exemplary view showing a state in which a stationary object is seen in accordance with a change in the position of a viewpoint;

4 is a view showing an example of determining the road surface conditions around the vehicle from the movement of the feature point in the present invention,

FIG. 5 is a view for explaining a method for calculating a depth of a portion corresponding to a feature point from the movement of the feature point representing a groove in the overhead view image according to the present invention; FIG.

6 is a view showing an example in which the determination result of the road surface state is reflected and displayed on an overhead view image according to the present invention;

7 is a view showing an example of a change in luminance when reflecting external light in a puddle, manhole, road surface,

8 is a diagram showing an example of detecting a road surface state based on image processing and vehicle behavior information in the present invention;

9 is a view showing an example of detecting the surrounding obstacles in the present invention and calling attention as needed,

FIG. 10 is a flowchart illustrating a process of determining a high and low portion of an actual road portion corresponding to a feature point by extracting and analyzing feature points in the present invention.

11 is a flowchart illustrating a process of determining whether a feature point is a puddle using luminance according to reflection of external light;

12 is a flow chart showing a process of accurately measuring the depth from the road surface based on the change in the stroke amount of the suspension in the present invention and alerting through the notification means,

FIG. 13 is a flowchart illustrating a process in which the notification means is activated or deactivated depending on whether or not vehicle stop position adjustment is possible when detecting obstacles and a pool (or groove) around the vehicle in the present invention. FIG.

<Explanation of symbols for the main parts of the drawings>

10: imaging means 11-14: camera

20: electronic processing device 21: image processing unit

30: notification means 31: display unit

32: speaker 40: vehicle information acquisition means

Claims (16)

delete Imaging means for photographing the surroundings of the vehicle; Vehicle information acquiring means for acquiring vehicle state information; Process the image photographed by the imaging means to generate an overhead view, extract a feature point from the generated overhead view, and then make a road surface around the vehicle based on the movement of the feature point and the vehicle state information acquired from the vehicle information acquisition means. An electronic processing apparatus for determining the three-dimensional object or the groove portion present on the image and acquiring and providing the result as road surface information; Notification means for outputting road surface information acquired and provided by the electronic processing apparatus and providing the same to a driver or a passenger; Including; The electronic processing apparatus searches for a position of a feature point extracted from an overhead view image one frame before, in the overhead view image of the current frame, and moves the feature point and the vehicle's movement direction, displacement amount or speed of the feature point one frame before and in the current frame, And analyzing the moving amount or the moving speed of the vehicle to determine the three-dimensional object or the groove portion present on the road surface around the vehicle. The method according to claim 2, The electronic processing apparatus, when the moving direction of the feature point and the moving direction of the vehicle is opposite to each other, the road surface information providing device around the vehicle, characterized in that the portion on the actual road corresponding to the feature point is determined to be a three-dimensional object higher than the surrounding road surface . The method according to claim 2, In the electronic processing apparatus, if the moving direction of the feature point and the moving direction of the vehicle are the same, and the displacement amount of the feature point and the movement amount of the vehicle or the moving speed of the feature point and the moving speed of the vehicle are different from each other, the actual road portion corresponding to the feature point is surrounded. A road surface information providing apparatus around a vehicle, characterized in that it is determined to be a groove portion lower than the road surface. The method according to claim 4, When the electronic processing apparatus determines the groove portion and determines that the front wheel or the rear wheel of the vehicle stops while passing through the groove portion, the suspension stroke amount measured by the vehicle information acquisition means when the front wheel or the rear wheel passes. And a guide according to the groove portion and the stop position along with the depth of the groove portion obtained from the change to the driver or passenger through the notification means. The method according to claim 5, And the electronic processing apparatus guides the driver to adjust the stop position of the vehicle upon notification through the notification means. The method according to claim 6, When the moving direction of the feature point and the moving direction of the vehicle are opposite to each other, the electronic processing apparatus determines that the portion on the actual road corresponding to the feature point is a three-dimensional object that is higher than the surrounding road surface, and the three-dimensional object is determined around the vehicle. And determining the stop position adjustment of the vehicle through the notification means if the stop position adjustment of the vehicle is impossible due to the three-dimensional object after determining the groove part. The method according to claim 5, When the electronic processing apparatus determines that the door lock is released through the door lock switch as the vehicle information acquiring means, the electronic processing apparatus provides a guide to warn the groove part when getting off through the in-vehicle notification means closest to the door. Informational device. The method according to claim 2, The electronic processing apparatus compares the moving direction of the feature point with the moving direction of the vehicle, and compares the displacement amount of the feature point with the moving amount of the vehicle or the moving speed of the feature point with the moving speed of the vehicle. A road surface information providing apparatus around the vehicle, characterized in that it is determined to have a height equal to the road surface. The method according to claim 9, The electronic processing apparatus determines whether the feature portion is a puddle by detecting the luminance according to the reflection of external light of the feature point assembly portion and the road surface portion in the desensitized image in the state where it is determined that the actual road portion corresponding to the feature point is equal to the road surface. A road surface information providing device around the vehicle, characterized in that. The method according to claim 10, And the electronic processing apparatus calculates a luminance ratio between the feature point set portion and the road surface portion in the overhead view image, and determines that it is a puddle when the luminance ratio is larger than a set value. The method according to claim 10, When the electronic processing apparatus determines the feature point portion as a pool and determines that the front wheel or the rear wheel of the vehicle stops after passing through the pool, the amount of suspension stroke measured by the vehicle information acquisition means when the front wheel or the rear wheel passes. A road surface information providing apparatus around a vehicle, characterized in that the driver or passenger is informed via the notification means, the guidance according to the pool and the stop position along with the depth of the pool obtained from the change. The method according to claim 12, And the electronic processing apparatus guides the driver to adjust the stop position of the vehicle upon notification through the notification means. 14. The method of claim 13, If the moving direction of the feature point and the moving direction of the vehicle are opposite to each other, the electronic processing apparatus determines that the portion on the actual road corresponding to the feature point is a three-dimensional object that is higher than the surrounding road surface, and the three-dimensional object is determined around the vehicle. And determining the stop position adjustment of the vehicle through the notification means if the stop position adjustment of the vehicle is impossible due to the three-dimensional object after the determination of the puddle. 14. The method of claim 13, When the electronic processing apparatus determines that the door lock is released through the door lock switch as the vehicle information acquisition means, the electronic processing apparatus provides a guide to pay attention to the puddle when getting off through the in-vehicle notification means closest to the door. Informational device. The method according to any one of claims 2 to 15, The electronic processing apparatus generates a top view overhead view image from the image photographed by the imaging means, synthesizes the replacement image of the vehicle on the generated overhead view, and displays the acquired road surface information to the notification means. It includes an image processing unit for providing, And the notifying means includes an output unit for receiving and outputting a final overhead image displayed together with the vehicle and road surface information around it from the image processing unit.

Images