KR102402087B1 - Parking assistance apparatus and control method for the same - Google Patents

Abstract

The present invention relates to a parking assistance apparatus and a control method thereof, and the method for controlling the parking assistance apparatus according to an embodiment of the present invention includes generating a first surrounding image of a first coordinate system using a camera mounted on a vehicle. step; converting the first surrounding image into a second surrounding image of a second coordinate system; recognizing a parking line from the second surrounding image; generating a first parking assistance line of the second coordinate system based on the parking line; converting the first parking assistance line into a second parking assistance line of the first coordinate system; and generating a parking assist image by synthesizing the second parking assist line with the first surrounding image. and outputting the parking assistance image.

Description

Parking assist device and its control method

The present invention relates to a parking assist device and a control method thereof, and more particularly, to a part corresponding to an actual parking line reflected in an image due to an error occurring in a mounting angle or mounting position of a camera when a vehicle is parked and parking A parking assist device for eliminating mismatch between auxiliary lines and a method for controlling the same.

A vehicle refers to a device that transports people, cargo, etc. from one place to another. Recently, as part of technology development to increase driver's convenience, a parking assist device has been developed and has reached the commercialization stage.

The parking assist device indicates whether parking is possible by measuring the length of a parking space using at least one camera or an ultrasonic sensor, or outputs an alert about a dangerous situation to the driver using an acoustic signal or an optical signal if necessary do. In addition, in some cases, it is responsible for steering and speed control for parking.

When such a parking assist device is activated, for example, a rear image of the vehicle may be output on a display provided in the vehicle, and at the same time, a parking assist line for guiding a parking path of the vehicle may be overlaid.

On the other hand, a camera that is mounted on a vehicle and generates a surrounding image to be output when parking may be deviated from the initially designed accurate mounting position or mounting angle due to various causes such as aging of the vehicle or external force.

In this case, as shown in FIG. 1 , a discrepancy area 13 between the actual parking line 11 in the image and the parking assist line 12 to guide the driver may be formed.

However, the conventional parking assistance apparatus has a problem that, even if an alignment error of the camera occurs, only the parking assistance line having a fixed position, shape, and size is always displayed, thereby adversely affecting the convenience and safety of the driver.

The present invention has been devised to solve the above problems, and according to embodiments of the present invention, when the vehicle is parked, due to an error occurring in the mounting angle or mounting position of the camera, the actual parking line and parking information reflected in the image An object of the present invention is to provide a parking assist device for eliminating inconsistencies between shipbuilding and a method for controlling the same.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention to achieve the above or other object, generating a first surrounding image of a first coordinate system using a camera mounted on a vehicle, and converting the first surrounding image into a second of a second coordinate system converting into a surrounding image, recognizing a parking line from the second surrounding image, generating a first parking assist line of the second coordinate system based on the parking line, and setting the first parking assist line converting to a second parking assist line of a first coordinate system, generating a parking assist image by synthesizing the second parking assist line with the first surrounding image, and outputting the parking assist image, The generating of the first parking assistance line of the second coordinate system includes calculating the shape, position and size of the parking line and the first parking having a shape, position and size corresponding to each of the shape, position and size of the parking line. generating an auxiliary line.

In addition, the camera may be mounted on one rear side of the vehicle, and the first surrounding image may be a rear image of the vehicle.

Also, the second surrounding image may be a bird's eye view image of the first surrounding image.

In addition, the step of recognizing a parking line and a parking line from the second surrounding image may include: detecting at least two or more edges from the second surrounding image; and recognizing, among the two or more edges, two or more edges matching a previously stored parking line pattern as the second parking line.

delete

The effects of the parking assist device and the control method thereof according to the present invention will be described as follows.

According to at least one of the embodiments of the present invention, it is possible to remove the discrepancy between the actual parking line and the parking assist line reflected in the image due to an error occurring in the mounting angle or mounting position of the camera when the vehicle is parked.

Accordingly, the driver's confusion can be reduced and the vehicle can be accurately parked in the parking position targeted by the driver, and in particular, it is important to prevent a situation in which the vehicle collides with an obstacle existing around it. can help

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a diagram referenced to explain the problems of the prior art.
2 shows a block diagram of a parking assistance device according to an embodiment of the present invention.
3 is a flowchart illustrating a method for controlling a parking assistance device according to an embodiment of the present invention.
4A and 4B are diagrams referenced for explaining an operation of detecting a parking line from a surrounding image by the parking assist device according to an embodiment of the present invention.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numbers regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "part" for the components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including an ordinal number such as 1st, 2nd, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When an element is referred to as being “connected” or “connected” to another element, it is understood that it may be directly connected or connected to the other element, but other elements may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as “comprises” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

2 shows a block diagram of a parking assistance apparatus 100 according to an embodiment of the present invention.

Referring to FIG. 2 , the parking assistance apparatus 100 according to an embodiment of the present invention includes a sensor unit 110 , an information storage unit 120 , a parking line detection unit 130 , and a parking assistance line generation unit 140 . , an image synthesizing unit 150 and an interface unit 160 .

The sensor unit 110 detects various objects existing around the vehicle, and outputs a sensing signal corresponding thereto. In this case, the sensing signal may be a signal output by the sensor unit 110 as a sensing result for the sensing region.

The sensor unit 110 includes at least one sensor. Specifically, the sensor unit 110 may include at least one of the cameras 112a - 112d and the ultrasonic sensor 114 . However, the type of sensor is not limited thereto, and if it is a sensor capable of detecting other objects, such as radar or lidar, it is not particularly limited.

The cameras 112a - 112d may generate an image of the surroundings by photographing the surroundings of the vehicle.

For example, any one of the cameras 112a - 112d 112a may be mounted on one side of the front of the vehicle to generate a front image of the vehicle.

For another example, the cameras 112a-112d may be sequentially mounted on one side of each of the front, rear, left, and right sides of the vehicle to generate a front image, a rear image, a left image and a right image, and the plurality of images are It may be synthesized by the image synthesizing unit 150 to be described later. By this synthesizing process, an around-view image as if looking down the vehicle from top to bottom may be generated.

As another example, when the gear of the vehicle is placed in the parking position, the rear camera 112b among the plurality of cameras 112a - 112d may be activated to generate a rear image of the vehicle.

Meanwhile, although 4 cameras 112a - 112d are illustrated in FIG. 2 , it should be understood that smaller or more cameras may be included in the sensor unit 110 .

The ultrasonic sensor 114 detects an obstacle around the vehicle and outputs a sensing signal corresponding to the detected obstacle. For example, a sensing signal corresponding to a second time point at which the ultrasound signal emitted at a first time point is reflected by an obstacle and returns, and a waveform or intensity of the reflected signal may be output.

The information storage unit 120 stores various information related to the parking area and the vehicle.

Specifically, the information storage unit 120 may store coordinate transformation information. The coordinate transformation information is information for transforming one coordinate system into another coordinate system. Such coordinate transformation information may be, for example, in a matrix format.

For example, the surrounding images generated by the cameras 112a - 112d may be represented by a two-dimensional first coordinate system, and the information storage unit 120 is a coordinate transformation matrix for converting the first coordinate system into a second coordinate system. can be saved. The second coordinate system may be a two-dimensional coordinate system corresponding to a bird's eye view. Accordingly, any one point in the first coordinate system may be projected to any one point in the second coordinate system, and conversely, any one point in the second coordinate system may be projected to any one point in the first coordinate system.

Also, the information storage unit 120 may permanently or temporarily store the sensing signal output by the sensor unit 110 .

In addition, the information storage unit 120 includes various programs for the operation of the parking line detection unit 130 , the parking auxiliary line generation unit 140 , the image synthesis unit 150 , and the interface unit 160 or input/output data. You can also save it.

In addition, the information storage unit 120, according to the request of the parking line detection unit 130, the parking auxiliary line generation unit 140, the image synthesis unit 150, and the interface unit 160 to be described later, at least among various pre-stored information A portion may be provided to the parking line detecting unit 130 , the parking auxiliary line generating unit 140 , the image synthesizing unit 150 , and the interface unit 160 .

Also, the information storage unit 120 may store in advance an alarm message for each situation to be output by the interface unit 160, which will be described later.

Also, the information storage unit 120 may store internal parameters and external parameters for each camera 112a - 112d.

The information storage unit 120 includes a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), and an SSD type. Various types of storage such as (Solid State Disk type), flash information storage unit 120 type (flash memory type), hard disk type, card type information storage unit 120, magnetic disk, optical disk, etc. It may include at least one of the media.

The parking line detector 130 may recognize at least one or more parking lines from surrounding images provided from the cameras 112a - 112d.

The parking line detection unit 130 may include a conversion module 132 , an edge detection module 134 , and a parking line detection module 136 .

The conversion module 132 may convert the coordinate system in which the surrounding images generated by the cameras 112a - 112d are handled into another coordinate system. For example, the cameras 112a - 112d generate a first peripheral image of a first coordinate system, and the transformation module 132 converts the coordinate values of each pixel of the first peripheral image to a second coordinate value different from the first coordinate system. A second surrounding image may be generated by converting the coordinate values of the coordinate system. That is, the transformation module 132 may convert the first peripheral image of the first coordinate system into the second peripheral image of the second coordinate system. Hereinafter, for convenience of description, it is assumed that the second coordinate system is a plane coordinate system corresponding to the buzz-eye view.

The edge detection module 134 detects at least two or more edges from the second peripheral image of the second coordinate system. For example, a gray scale of each of the pixels in the second peripheral image may be calculated, and edges forming a straight line may be detected based on the calculated gray scale.

The parking line detection module 136 may recognize, among the edges detected by the edge detection module 134 , edges having a preset parking line pattern as the aforementioned actual parking lines.

The parking assist line generating unit 140 generates a parking assist line for guiding the corresponding parking line based on the parking line recognized by the parking line detecting unit 130 . For example, the parking line detection unit 130 recognizes the aforementioned parking line from the second surrounding image of the second coordinate system, and the parking assistance line generator 140 generates the first parking assistance line of the second coordinate system, This may be converted into a second parking assist line of the first coordinate system.

Meanwhile, the shape, position, and size of the first parking assistance line may correspond to the shape, position, and size of the parking line recognized by the parking line detection unit 130 , respectively.

The image synthesizing unit 150 may generate a parking assist image by synthesizing the second parking assist line with the above-described first surrounding image. Since both the second parking assist line and the first surrounding image are expressed in the first coordinate system, a parking assist image may be generated by synthesizing the two.

The interface unit 160 outputs various types of information related to a vehicle or a parking section as signals in a form that can be recognized by the user, or receives various commands from the driver.

In addition, the interface unit 160 may receive a parking command, an exit command, a steering command, an acceleration command, a deceleration command, and the like from the driver.

To this end, the interface unit 160 may include a sound module 162 and a display module 164 .

The sound module 142 outputs various messages as an audible signal using a speaker provided therein, receives the driver's voice command using a microphone, etc. and converts it into an electric signal corresponding thereto, and then includes a parking line detection unit ( 130 ), the parking assist line generating unit 140 , the image synthesizing unit 150 , and the interface unit 160 may be provided.

The display module 164 may output various messages as visual signals using a head-up display (HUD) provided therein. In addition, the display module 144 may display the parking assistance image generated by the image synthesizing unit 150 .

In addition, the interface unit 160 receives various commands, such as a driver's touch, by using a touch sensor or button provided therein, and converts it into an electric signal corresponding thereto, and then the parking line detection unit 130 and the parking assist line generation unit. 140 , the image synthesis unit 150 , and the interface unit 160 may provide at least one of them.

Meanwhile, the parking assistance apparatus 100 described above with reference to FIG. 2 may be mounted on a vehicle.

3 is a flowchart illustrating a control method of the parking assistance apparatus 100 according to an embodiment of the present invention. For better understanding, an image generated by the rear camera 112d among the plurality of cameras 112a to 112d shown in FIG. 2 will be described as a reference.

Referring to FIG. 3 , the camera 112d generates a first surrounding image of the first coordinate system ( S310 ). For example, the first coordinate system may be a camera coordinate system, and in this case, the first surrounding image may be an original image captured by the camera 112d at a mounting position and a mounting angle of the camera 112d.

Next, the parking line detection unit 130 converts the first surrounding image into a second surrounding image of the second coordinate system (S320). For example, in step S320, the first surrounding image may be converted into a second surrounding image of a two-dimensional planar coordinate system corresponding to the buzz-eye view. In this case, the parking line detection unit 130 may convert the first surrounding image into the second surrounding image by using the first coordinate transformation matrix stored in the information storage unit 120 . Here, the first coordinate transformation matrix is a matrix for transforming the first coordinate system into the second coordinate system. Accordingly, any one point in the first coordinate system may be matched one-to-one with any one point in the second coordinate system.

Next, the parking line detection unit 130 may recognize the parking line from the second surrounding image (S330). When the second surrounding image is the buzz-eye view, a parking line closer to the real one may appear compared to the first surrounding image. A method of recognizing a parking line appearing in the first surrounding image without a conversion process to the second surrounding image In comparison, parking line recognition accuracy may be improved.

Next, the parking assistance line generator 140 may generate a first parking assistance line of the second coordinate system based on the parking line of the second coordinate system recognized through step S330 ( S340 ). Specifically, after calculating the shape, position, and size of the parking line recognized through step S330, the parking assist line generating unit 140 has a shape, position, and size corresponding to each of the calculated shape, position, and size of the parking line. A first parking assist line may be generated. That is, even if an error occurs in the mounting position and mounting angle of the camera 112d, through step S340, the first parking assist line reflecting the alignment error of the camera 112d is generated, and thus the parking line and the parking assist line according to the prior art The problem of inconsistency between the two can be eliminated.

Next, the parking assistance line generator 140 may convert the first parking assistance line of the second coordinate system into the second parking assistance line of the first coordinate system ( S350 ). In this case, the parking assistance line generator 140 may convert the first parking assistance line into the second parking assistance line by using the second coordinate transformation matrix stored in the information storage unit 120 . Here, the second coordinate transformation matrix is a matrix for transforming the second coordinate system into the first coordinate system. Accordingly, any one point in the second coordinate system may be matched one-to-one with any one point in the first coordinate system. Meanwhile, the second coordinate transformation matrix may be an inverse matrix of the first coordinate transformation matrix described above in the description of step S320.

Next, the image synthesizing unit 150 generates a parking assist image by synthesizing the first surrounding image generated in step S310 and the second parking assist line generated in step S350 ( S360 ). Since both the first surrounding image and the second parking assistance line are expressed in the first coordinate system, which is the same coordinate system, a separate coordinate transformation process may not be required when combining the two.

Meanwhile, the synthesis between the first surrounding image and the second parking assistance line in step S360 may be performed by overlaying the second parking assistance line on the first surrounding image. As described above, since the first parking assist line has characteristics corresponding to the shape, position, and size of the parking line appearing in the second surrounding image, the converted second parking assist line also has the shape of the parking line appearing in the first surrounding image. , may correspond to the location and size.

In addition, when synthesizing the first surrounding image and the second parking assist line, the image synthesizing unit 150 may display the second parking assist line at the location of the parking line appearing in the first surrounding image.

Next, the display module 164 may output the parking assistance image generated through step S360 (S370). When the parking assist image is displayed, any one of the parking line and the second parking assist line appearing in the first surrounding image may be overlaid on the other.

4A and 4B are diagrams referenced for explaining an operation of detecting a parking line from a surrounding image by the parking assistance apparatus 100 according to an embodiment of the present invention.

First, FIG. 4A (a) illustrates a top view when the center line C1 of the vehicle 400 and the center line C2 of the parking section formed of the parking line 410 coincide. The camera 112d may capture a predetermined angular range behind the vehicle 400 to generate a surrounding image corresponding thereto.

Meanwhile, errors may exist in the mounting position and mounting angle of the camera 112d mounted on the vehicle 400 as shown in (a) of FIG. 4A . For example, if there is no error in the mounting position and angle of the camera 112d, the first range S1 symmetrical with respect to the center line C1 of the vehicle is normally photographed, but when an alignment error occurs, the first range ( The second range S2 rotated from S1 by a predetermined angle θ may be photographed.

In this case, even if the center line C1 of the vehicle and the center line C2 of the parking section coincide, the first peripheral image 420 of the first coordinate system generated by the camera 112d is displayed through the display module 164 In this case, as shown in (b) of FIG. 4A , the graphic object 421 corresponding to the actual parking line 410 is not symmetrical about the vertical central axis C3 of the image 420 and is skewed to one side. may appear For this case, the parking assist line for guiding the graphic object 421 corresponding to the actual parking line 410 (see reference numeral 451 of FIG. 4B (a) to be described later) is also the vertical central axis of the image 420 ( It needs to be displayed in a form skewed to one side around C3).

Next, FIG. 4B is a diagram referenced to explain an operation of generating a parking assistance image by using the first surrounding image 420 generated in the situation of FIG. 4A (a).

The parking line detector 130 converts the first peripheral image 420 of the first coordinate system with respect to the second range S2 to the second peripheral image 430 of the second coordinate system as shown in (a) of FIG. 4B . can In the process of converting the first surrounding image 420 into the second surrounding image 430 , a portion of the first surrounding image 420 may be excluded from the conversion target.

When the second coordinate system is a plane coordinate system corresponding to the bird's eye view, a graphic object 422 corresponding to the actual parking line 410 appears in the second surrounding image 430 as shown in (a) of FIG. 4B . .

The parking line detection unit 130 detects a straight edge by performing edge detection on the second surrounding image 430, and since all four corners constituting the actual parking line 410 are straight, a graphic object ( Since the edges corresponding to the four corners constituting 431 also have a straight shape, the parking line detection unit 130 may recognize the graphic object 431 as a parking line.

Meanwhile, the parking assistance line generator 140 may calculate information on the parking line 431 recognized in (a) of FIG. 4B . For example, the parking assist line generator 140 may calculate the recognized width W1 , the length L1 , the slope α , and the position P1 of the parking line 431 .

Here, the inclination α may be an angle between the rectangular center line C5 of the parking line 431 and the vertical central axis C4 of the second peripheral image 430 . Also, the position P1 of the rectangle formed by the parking line 431 may be a center position of the rectangle formed by the parking line 431 .

Next, the parking assistance line generator 140 may generate a first parking assistance line 441 of the second coordinate system based on the information on the parking line 431 . Referring to FIG. 4B (b), the first parking assistance line 441 corresponds to each of the width W1, the length L1, the slope α, and the position P1 of the rectangle made of the parking line 431 . may have a width W2, a length L2, a slope β, and a position P2. Here, the slope β may be an angle formed by the rectangular center line C6 of the first parking assistance line 441 and the vertical central axis C4 of the second peripheral image 430 . In addition, the position P2 of the rectangle formed by the first parking assistance line 441 may be a center position of the rectangle formed by the first parking assistance line 441 . At this time, the width W1, the length H1, the slope α and the position P1 of the rectangle made of the parking line 431 and the width W2, the length L2 of the first parking auxiliary line 441, The slope β and the position P2 may each be the same.

The parking assistance line generator 140 generates a first parking assistance line 441 of the second coordinate system shown in (b) of FIG. 4B, and a second parking assistance line 441 of the second coordinate system shown in FIG. It can be converted into a parking assist line 451 .

Next, the image synthesizing unit 150 synthesizes the second parking assist line 451 shown in (c) of FIG. 4B with the first surrounding image 420 shown in (b) of FIG. 4A to assist parking. You can create an image. In this case, the second parking assistance line 451 has a shape, position, and size corresponding to the parking line 421 appearing in the first surrounding image 420 .

In addition, when the parking assistance image is displayed through the display module 164, the image synthesis unit 150 overlays any one of the second parking assistance line 451 and the parking line 421 on the other. have.

The embodiment of the present invention described above is not implemented only through the apparatus and method, and may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded. The implementation can be easily implemented by those skilled in the art to which the present invention pertains from the description of the above-described embodiments.

In addition, the present invention described above, various substitutions, modifications and changes are possible within the scope that does not depart from the technical spirit of the present invention for those of ordinary skill in the art to which the present invention pertains, so the above-described embodiment and It is not limited by the accompanying drawings, and all or part of each embodiment may be selectively combined so that various modifications may be made.

100: parking aid
110: sensor unit
120: information storage unit
130: parking line detection unit
140: parking aid line generation unit
150: image synthesis unit
160: interface unit

Claims (5)

generating a first surrounding image of a first coordinate system using a camera mounted on a vehicle;
converting the first surrounding image into a second surrounding image of a second coordinate system;
recognizing a parking line from the second surrounding image;
generating a first parking assistance line of the second coordinate system based on the parking line;
converting the first parking assistance line into a second parking assistance line of the first coordinate system;
generating a parking assist image by synthesizing the second parking assist line with the first surrounding image; and
outputting the parking assistance image;
including,
The step of generating the first parking assistance line of the second coordinate system comprises:
calculating the shape, position and size of the parking line; and
generating the first parking assist line having a shape, position, and size corresponding to each of the shape, position, and size of the parking line;
A control method of a parking assist device comprising a. The method of claim 1,
The camera is mounted on the rear side of the vehicle,
The first surrounding image is a rear image of the vehicle, the control method of the parking assist device. The method of claim 1,
The second surrounding image,
A method of controlling a parking assist device, which is a bird's eye view image for the first surrounding image. The method of claim 1,
Recognizing a parking line from the second surrounding image comprises:
detecting at least two or more edges from the second surrounding image; and
recognizing, among the two or more edges, two or more edges matching a previously stored parking line pattern as the second parking assist line;
A control method of a parking assist device comprising a. delete

Images