KR20220089788A - Method And Apparatus for Parking Assistance Using Camera for Vehicle - Google Patents

Abstract

Disclosed is a method and apparatus for parking assistance using a vehicle camera.
According to an embodiment of the present disclosure, in a parking assistance method using a vehicle camera (camera), a plurality of wide angle cameras mounted on the vehicle (wide angle camera) to photograph a space around the vehicle; A process in which a plurality of depth cameras mounted on the vehicle take a depth map around the vehicle; A process of detecting a parking space around a vehicle by identifying a parking line based on a surround view image obtained from a plurality of wide-angle cameras; The process of determining whether the size of the parking space is the size of the vehicle can be parked; a process of detecting an obstacle in a parking space based on a depth image in the parking space obtained from a depth camera when the size of the vehicle is available for parking; and determining whether the parking space is an area in which a vehicle can be parked based on whether obstacles are detected.

Description

Method And Apparatus for Parking Assistance Using Camera for Vehicle

The present disclosure relates to a method and apparatus for parking assistance using a vehicle camera. More specifically, it relates to a parking assistance method and apparatus using a depth camera.

The content described in this section merely provides background information for the present disclosure and does not constitute the prior art.

In order to increase the safety and convenience of a driver using a vehicle, the development of a technology for grafting various sensors and electronic devices to a vehicle is accelerating. Accordingly, autonomous driving in which a vehicle can travel on a road by itself without the intervention of a driver has become possible.

Meanwhile, interest in self-parking technology that automatically performs parking, which is difficult for many drivers, among autonomous vehicles, is also increasing. In general, in order to support automatic parking, an image of a camera or an ultrasonic wave sensor is used to search for a parking space around the vehicle, and the searched parking space is guided to the driver.

In the case of using an ultrasonic sensor, a method of measuring the inter-vehicle distance from the round-trip time it takes for the emitted ultrasonic waves to collide with an object and return again is used. A method of recognizing a parking space based on a surround view is used.

However, in the case of automatic parking using ultrasonic waves, the inter-vehicle distance can be easily measured, but it is difficult to accurately recognize a lane, and thus correct parking is difficult. In addition, even in the case of automatic parking using the image of the camera, there is a problem in that it is difficult to detect if there is a 3D obstacle in the parking space because the space is detected based on 2D (Dimension).

Accordingly, the present disclosure is intended to solve these problems, and while utilizing the camera mounting space of an existing vehicle as it is, using a depth camera to recognize a parking space as well as to provide a parking assist method and device for more accurately detecting obstacles in a parking space The main purpose is to

According to an embodiment of the present disclosure for achieving this object, in the parking assistance method using a vehicle camera (camera), a plurality of wide angle cameras mounted on the vehicle take a picture of the space around the vehicle. process; a process in which a plurality of depth cameras mounted on the vehicle take a depth map around the vehicle; detecting a parking space around the vehicle by identifying a parking line based on a surround view image obtained from the plurality of wide-angle cameras; a process of determining whether the size of the parking space is a size in which the vehicle can be parked; detecting an obstacle in the parking space based on the depth image in the parking space obtained from the depth camera when the size of the vehicle is available for parking; and determining whether the parking space is an area in which parking of the vehicle is possible based on whether obstacles are detected.

In addition, in the parking assist device using a vehicle camera, a plurality of wide-angle cameras mounted on a vehicle and configured to photograph a space around the vehicle; a plurality of depth cameras mounted on the vehicle and configured to capture depth images around the vehicle; a parking space detection unit for detecting a parking space around the vehicle by identifying a parking line based on the surround view images obtained from the plurality of wide-angle cameras; an obstacle detection unit configured to detect an obstacle around the vehicle based on the depth image in the parking space obtained from the plurality of depth cameras; and a parking availability determining unit that determines whether the vehicle can be parked based on the size of the parking space and whether obstacles are detected.

As described above, according to the present embodiment, it is possible to assist parking by not only recognizing a parking space but also more accurately detecting obstacles in the parking space while utilizing the camera mounting space of the existing vehicle as it is.

1 is a block diagram of a parking assistance device according to an embodiment of the present disclosure.
2 is a perspective view of a parking assist device according to an embodiment of the present disclosure.
3 is a view for explaining a parking space detection process according to an embodiment of the present disclosure.
4 is a view for explaining an obstacle detection process according to an embodiment of the present disclosure.
5 is a flowchart illustrating a parking assistance method according to an embodiment of the present disclosure.
FIG. 6 is a flowchart illustrating in detail the process S120 of FIG. 5 .
7 is a flowchart illustrating in detail the process S140 of FIG. 5 .

Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. In adding reference numerals to components in each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated in different drawings. In addition, in describing the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description thereof will be omitted.

In describing the components of the embodiment according to the present disclosure, reference numerals such as first, second, i), ii), a), b) may be used. These signs are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the signs. In the specification, when a part 'includes' or 'includes' a certain element, it means that other elements may be further included, rather than excluding other elements, unless explicitly stated otherwise. .

1 is a block diagram of a parking assistance device according to an embodiment of the present disclosure.

2 is a perspective view of a parking assist device according to an embodiment of the present disclosure.

1 and 2 , the parking assist device according to an embodiment of the present disclosure includes a wide angle camera 100, a depth camera 110, a surround view generation unit, 120), a depth map generation unit 130, a parking space detection unit 140, an obstacle detection unit 150, a parking availability determining unit 160 ), a display unit (170) and an alarm unit (alarm unit, 180) includes all or a part.

The wide-angle camera 100 is mounted on a vehicle and configured to photograph a space around the vehicle. A plurality of wide-angle cameras 100 may be configured to be mounted on a vehicle, and a plurality of wide-angle cameras 100 may detect a parking space by photographing a space around the vehicle.

The wide-angle camera 100 may include a lens module (not shown) and a circuit board (circuit board, not shown), and the lens module includes a plurality of lenses (lens, not shown) including a lens barrel, It means that it is assembled in (not shown). Meanwhile, the circuit board may include an image sensor (not shown) configured to convert light incident through the lens module into an electrical signal.

The depth camera 110 is mounted on the vehicle and is configured to take a depth image around the vehicle. A plurality of depth cameras 110 may be configured to be mounted on a vehicle, and the plurality of depth cameras 110 may detect an obstacle by photographing a depth image around the vehicle.

The depth camera 110 may be configured as, for example, a TOF (Time of Flight) camera, and the TOF camera illuminates an object to be photographed with infrared rays, receives infrared rays reflected back from the object, and calculates the time it takes to return. , can measure the distance to an object. Since the depth camera 110 takes a depth image using infrared rays, unlike a general camera, it is possible to detect an obstacle even at night, and compared to the case of detecting an obstacle using an ultrasonic wave, the obstacle detection distance is much longer, and the obstacle The size and position of the can be detected more accurately.

When the depth camera 110 is configured as a TOF camera, the depth camera 110 may include an infrared lighting unit 200 and an infrared receiving unit 210 .

The infrared illuminator 200 is configured to irradiate infrared rays. At this time, infrared irradiation may be made toward the front. The infrared illuminator 200 may be formed of an infrared LED (Light Emitting Diode), but is not necessarily limited thereto as long as infrared radiation is possible. Meanwhile, the infrared ray irradiated from the infrared illuminator 200 is preferably configured to have a wavelength of about 940 nm in order to reduce interference from sunlight, but is not necessarily limited thereto.

The infrared light receiving unit 210 is configured to receive the infrared ray irradiated from the infrared illuminator 200 is reflected back by hitting an object. The infrared light receiving unit 210 may generate an electrical signal when receiving infrared rays, and accordingly, it is possible to measure the time until the infrared rays are irradiated from the infrared lighting unit 200 and return to the infrared light receiving unit 210 .

Meanwhile, as shown in FIG. 2 , the depth camera 110 and the wide-angle camera 100 may be integrally configured to form one module. In this case, the module including the depth camera 110 and the wide-angle camera 100 may be mounted on, for example, a side mirror of a vehicle, a radiator grill, a trunk, and the like. At this time, it is preferable that the module including the depth camera 110 and the wide-angle camera 100 is configured to be mounted as it is in a portion where the camera of an existing vehicle is mounted.

The surround view generating unit 120 generates a surround view image by synthesizing and signal processing images captured by the plurality of wide-angle cameras 100 . The plurality of wide-angle cameras 100 may each be mounted on a vehicle's side mirror, radiator grille, trunk, etc. In this case, the surround view image may be an image corresponding to a virtual viewpoint, for example, a viewpoint viewing the vehicle from above. .

The depth image generator 130 generates a depth image, which is a 3D stereoscopic image, by using the image captured by the depth camera 110 . The depth image may represent an image containing distance information from the photographed object surface. For example, distance information may be indicated by expressing a surface near from the depth camera 110 in a bright color and a surface far from the depth camera 110 in a dark color. Accordingly, it is possible to easily detect obstacles around the vehicle and the distance to the obstacles using the depth camera 110 . In addition, not only can an obstacle be detected using the depth camera 110 , but also the presence or absence of a sink hole can be detected.

The parking space detection unit 140 detects a parking space around the vehicle by identifying a parking line based on the surround view image obtained from the plurality of wide-angle cameras 100 . Here, the parking space means a rectangular space existing on the ground, and the rectangular space means a space inside the parking line. In this case, the rectangular space may be detected based on four points at which parking lines are identified in the surround view image and the identified parking lines intersect.

For detection of a parking space, the parking space detection unit 140 may include a contour extraction unit 142 and a parking line detection unit 144 .

In addition, although not shown in FIG. 1 , the parking space detection unit 140 includes a distortion correction unit (not shown) for correcting a curved line that may appear distorted in the process of converting a 3D stereoscopic image into a 2D image. city) may be further included.

The outline extraction unit 142 extracts the outline for the parking space. For example, from the image corrected by the distortion correction unit, contours for the vehicle and the parking line may be extracted using, for example, a Canny edge detection method.

The parking line identification unit 144 identifies the parking line by applying, for example, Hough transform to the inside of the outline extracted by the outline extraction unit 142 . The Hough transform is one of the algorithms for finding a straight line in a video image, and by using it, a straight line inside the outline can be extracted, and the extracted straight line can be identified as a parking line.

The obstacle detecting unit 150 detects obstacles around the vehicle based on the depth image in the parking space obtained from the plurality of depth cameras 110 . That is, when there is an obstacle inside the parking space detected by the parking space detection unit 140, the color of the part with the obstacle in the depth image is displayed differently from the surroundings, and the obstacle detecting unit 150 detects the obstacle in the parking space based on this. presence can be detected. Also, the obstacle detecting unit 150 may detect the presence or absence of a sink hole in the same manner as above.

For obstacle detection, the obstacle detection unit 150 may include a histogram calculation unit 152 and a pixel comparison unit 154 .

The histogram calculation unit 152 calculates a histogram of the depth image in the parking space. For example, the histogram calculation unit 152 may divide the parking space at regular intervals to calculate a histogram of pixel values for each divided area, and calculate how much the histogram differs between the divided areas. Based on the difference in the histogram, the obstacle detecting unit 150 may detect an obstacle existing in the parking space.

The pixel comparison unit 154 compares the similarity between neighboring pixels by using vertical and horizontal scanning of the depth image in the parking space. For example, when there is an obstacle in the depth image, a large difference appears between the pixels in the area with the obstacle compared with the neighboring pixels, and this large difference can be recognized by scanning the pixel values vertically and horizontally. Accordingly, the obstacle detecting unit 150 may detect an obstacle in the parking space based on the similarity between neighboring pixels.

The parking possibility determination unit 160 determines whether the vehicle can be parked based on the size of the parking space and whether obstacles are detected.

The parking possibility determination unit 160 first determines whether the size of the parking space detected by the parking space detection unit 140 is a size in which a vehicle can be parked. Accordingly, the parking possibility determination unit 160 may store information on the size of the vehicle the driver is riding in, and accordingly, the rectangular parking space detected by the parking space detection unit 140 is sufficiently large compared to the size of the vehicle. If it is not large, it may be determined that parking is impossible.

In addition, the parking possibility determination unit 160 may determine whether the detected parking space is a parking area according to whether the obstacle detection unit 150 detects an obstacle. For example, even when the size of the parking space is sufficiently large compared to the size of the vehicle to be parked, if there is an obstacle or a sink hole in the parking space, parking may not be possible. It can be considered as an impossible area.

However, even when the obstacle and/or the sink hole sensed by the obstacle detection unit 150 is detected, the parking possibility determination unit 160 determines that the position of the obstacle and/or the sink hole is, for example, at the edge of the parking space. When a sufficient parking space is secured despite the presence of an obstacle and/or a sinkhole, it may be determined that the parking space is available. In this case, the position of the obstacle and/or the sinkhole may be determined based on the calculation and comparison of the histogram calculator 152 and the pixel comparison unit 154 .

The display unit 170 is provided inside the vehicle, and when the parking possibility determination unit 160 determines that parking is possible, the display unit 170 is configured to display a parking possibility to the driver. The display unit 170 may be provided in a vehicle cluster, a center fascia, or the like, but is not necessarily limited to the above position as long as the driver can see the parking available indication.

The alarm unit 180 is provided inside the vehicle, and when the parking possibility determination unit 160 determines that parking is possible, it is configured to sound a parking available notification to the driver.

The parking assistance device according to an embodiment of the present disclosure may assist parking by using a vehicle camera and including the above configuration to more accurately detect obstacles in a parking space as well as recognize a parking space.

3 is a view for explaining a parking space detection process according to an embodiment of the present disclosure.

Referring to FIG. 3 , in the parking space detection process according to an embodiment of the present disclosure, a surround view is generated based on an image using the wide-angle camera 100 as shown in FIG. 3A . The generated surround view is converted into a 2D image as shown in (b) of FIG. 3 . In the conversion process, distortion correction can be made and the contour of the parking space is detected. After that, a Hough transform is applied to detect a straight line component in the image as shown in (c) of FIG. 3, and as shown in (d) and (e) of FIG. detect in space. If the image of the parking space is transformed into a projection (perspective transform), it is possible to obtain an image as shown in (f) of FIG. 3 . At this time, the thick border area shown in (f) of FIG. 3 becomes a parking space, and it can be determined whether parking is possible based on the size of the area.

On the other hand, although FIG. 3 shows the case of T-type parking or perpendicular parking, the parking space detection process according to an embodiment of the present disclosure is not necessarily limited to this case, and even in parallel parking, the above process is The same can be done.

4 is a view for explaining an obstacle detection process according to an embodiment of the present disclosure.

Referring to FIG. 4 , the histogram in the parking space calculated by the histogram calculation unit 152 may appear as shown in FIGS. 4A and 4B , and FIG. (b) of 4 shows the histogram distribution when there is an obstacle. For example, when there is an object indicating no parking in the parking space as shown in (c) of FIG. 4, a histogram distribution as shown in (b) of FIG. 4 may appear. Based on this histogram distribution, the obstacle detecting unit 150 may detect whether an obstacle exists in the parking space.

5 is a flowchart illustrating a parking assistance method according to an embodiment of the present disclosure.

Referring to FIG. 5 , in the parking assistance method according to an embodiment of the present disclosure, a plurality of wide-angle cameras 100 mounted on a vehicle photograph a space around the vehicle ( S100 ). A plurality of depth cameras 110 mounted on the vehicle take a depth image around the vehicle (S110). The detection process of the parking space is performed (S120), and the parking possibility determination unit 160 determines whether the detected size of the parking space is a size in which a vehicle can be parked, that is, whether parking is possible in the detected size of the parking space. (S130). If it is determined that parking is impossible due to the detected size of the parking space, the parking space detection process (S120) is continuously performed.

When it is determined that the size of the vehicle parking is possible, an obstacle detection process is performed (S140), and it is determined whether the detected parking space is an area where the vehicle can be parked based on whether the obstacle is detected (S150). When an obstacle and/or a sinkhole is detected and it is determined that it is an area in which parking is impossible, the parking space detection process ( S120 ) is continuously performed.

When it is determined that the detected parking space is a parking available area because an obstacle and/or a sinkhole is not detected, the driver starts parking or autonomous parking starts ( S160 ). In this case, the display unit 170 provided in the vehicle may indicate that parking is available to the driver, and the alarm unit 180 provided in the vehicle may sound a parking available alarm to the driver of the vehicle.

The parking assistance method according to an embodiment of the present disclosure includes the above process, so that not only the recognition of a parking space using the depth camera 110 but also an obstacle in the parking space can be more accurately detected to assist parking.

FIG. 6 is a flowchart illustrating in detail the process S120 of FIG. 5 .

Referring to FIG. 6 , the parking space detection process ( S120 ) may include the following process. A surround view image is obtained from a plurality of wide-angle cameras 100 (S121). In this case, the surround view image may be generated by the surround view generator 120 . Also, a process of removing noise from the generated surround view image may be performed ( S122 ). When the noise cancellation process is performed, pixels that are out of a predetermined limit from the average value can be removed, so that the sharpness of the image can be increased.

The outline extraction unit 142 extracts the outline of the parking space from the video image (S123). The parking line identification unit 144 applies the Hough transform to the inside of the outline (S124), and thus identifies the parking line (S125). Four points where the identified parking lines intersect are extracted (S126), and a rectangular parking space is detected based on this (S127).

The parking assistance method according to an embodiment of the present disclosure may include a parking space detection process as described above, thereby detecting a parking space around the vehicle.

7 is a flowchart illustrating in detail the process S140 of FIG. 5 .

Referring to FIG. 7 , the obstacle detection process ( S140 ) may include the following process. A depth image in the parking space is obtained from a plurality of depth cameras 110 (S141). In this case, the depth image may be generated by the depth image generator 130 . In addition, a process of removing noise from the generated depth image image may be performed (S200). When the noise removal process is performed, pixels out of a predetermined limit from the average value can be removed, so that the sharpness of the image can be increased.

The histogram calculation unit 152 calculates a histogram for the depth image in the parking space (S143). The pixel comparison unit 154 performs vertical and horizontal scanning of the depth image in the parking space (S210), and thus compares the similarity between neighboring pixels (S145). The presence or absence of an obstacle in the parking space is detected based on the calculated histogram and the similarity between neighboring pixels (S146).

The parking assistance method according to an embodiment of the present disclosure may detect an obstacle in a parking space by including the obstacle detection process as described above.

The above description is merely illustrative of the technical idea of this embodiment, and a person skilled in the art to which this embodiment belongs may make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present embodiment.

100: wide-angle camera 110: depth camera
120: surround view generator 130: depth image generator
140: parking space detection unit 200: outline extraction unit
210: parking line identification unit 150: obstacle detection unit
152: histogram calculation unit 154: pixel comparison unit
160: parking possibility determination unit 170: display unit
180: alarm unit 200: infrared lighting unit
210: infrared light receiving unit

Claims (13)

In the parking assistance method using a vehicle camera (camera),
A process of photographing a space around the vehicle by a plurality of wide angle cameras mounted on the vehicle;
a process in which a plurality of depth cameras mounted on the vehicle take a depth map around the vehicle;
detecting a parking space around the vehicle by identifying a parking line based on a surround view image obtained from the plurality of wide-angle cameras;
a process of determining whether the size of the parking space is a size in which the vehicle can be parked;
If the vehicle is of a size that can be parked,
detecting an obstacle in the parking space based on the depth image in the parking space obtained from the depth camera; and
The process of determining whether the parking space is an area in which parking of the vehicle is possible based on whether obstacles are detected
A parking assistance method comprising a. The method of claim 1,
The process of detecting a parking space around the vehicle,
extracting an outline for the parking space; and
A process of identifying a parking line by applying a Hough transform to the inside of the outline
A parking assistance method comprising a. The method of claim 1,
The process of detecting a parking space around the vehicle,
A parking assistance method, characterized in that it is achieved by detecting a rectangular parking space based on four points where the identified parking lines intersect. The method of claim 1,
The process of detecting obstacles around the vehicle,
calculating a histogram for the depth image in the parking space; and
The process of detecting an obstacle in the parking space based on the calculated histogram
A parking assistance method comprising a. The method of claim 1,
The process of detecting obstacles around the vehicle,
calculating a pixel similarity between neighboring pixels using vertical and horizontal scanning of the depth image in the parking space; and
The process of detecting an obstacle in the parking space based on the similarity between neighboring pixels
A parking assistance method comprising a. The method of claim 1,
If it is determined that the vehicle can be parked in an area,
a process in which a display unit provided inside the vehicle displays parking available; or
A process in which an alarm unit provided inside the vehicle sounds a parking available alarm to the driver of the vehicle
Parking assistance method, characterized in that it further comprises. In the parking assist device using a vehicle camera,
a plurality of wide-angle cameras mounted on a vehicle and configured to photograph a space around the vehicle;
a plurality of depth cameras mounted on the vehicle and configured to photograph a depth image around the vehicle;
a parking space detection unit for detecting a parking space around the vehicle by identifying a parking line based on the surround view images obtained from the plurality of wide-angle cameras;
an obstacle detection unit configured to detect an obstacle around the vehicle based on the depth image in the parking space obtained from the plurality of depth cameras; and
A parking availability determining unit that determines whether the vehicle can be parked based on the size of the parking space and whether an obstacle is detected
A parking assist device comprising a. 8. The method of claim 7,
The parking space detection unit,
a contour extraction unit for extracting a contour for the parking space; and
A parking line detection unit that identifies a parking line by applying a Hough transform to the inside of the outline extracted by the outline extraction unit
A parking assist device comprising a. 8. The method of claim 7,
The parking space detection unit,
A parking assist device, characterized in that it detects a rectangle based on four points where the identified parking lines intersect as a parking space. 8. The method of claim 7,
The obstacle detection unit,
Including a histogram calculation unit for calculating a histogram for the depth image in the parking space,
Parking assistance device, characterized in that for detecting an obstacle in the parking space based on the histogram calculated by the histogram calculation unit. 8. The method of claim 7,
The obstacle detection unit,
and a pixel comparison unit that calculates the similarity between neighboring pixels using vertical and horizontal scanning of the depth image in the parking space,
A parking assist device, characterized in that it detects an obstacle in the parking space based on the similarity between neighboring pixels. 8. The method of claim 7,
a display unit provided inside the vehicle and configured to display a parking availability indication to the driver when the parking possibility determination unit determines that parking is possible; and
An alarm unit provided inside the vehicle and configured to sound a parking available alarm to the driver when the parking possibility determination unit determines that parking is possible
Parking assist device, characterized in that it further comprises. 8. The method of claim 7,
The wide-angle camera and the depth camera are integrally configured to form a single module (module).

Images