KR102322815B1 - Electronic Device and the Method for Detecting Occupancy Lane of Road Event Based on Camera - Google Patents

Abstract

An apparatus for detecting a road event occupied lane according to the present invention includes: a camera for generating an image by photographing a road around a vehicle; a memory storing at least one instruction; and a processor, wherein, as the at least one instruction is executed, an image obtained by photographing a surrounding road of the vehicle from the camera is acquired, and a road event acting as a risk factor on the surrounding road from the image When the road event is recognized, the road event occupied lane occupied by the road event may be determined by comparing the angle between the reference line and the road event and the angle between the reference line and the lane in the image.
According to the present invention, a lane occupied by a road event occurring on a road around the vehicle can be easily detected using only images captured by a camera without the aid of a Global Navigation Satellite System (GNSS) or map information, thereby improving the autonomous driving of the vehicle. It can help a lot.

Description

Camera-based road event occupancy lane detection device and method

The present invention relates to a technology for detecting an occupied lane occupied by various road events that may occur on a road based on an image captured by a camera installed in a vehicle without the aid of a Global Navigation Satellite System (GNSS) or map information.

Research on autonomous vehicles is being actively conducted. The autonomous vehicle may recognize a lane and surrounding objects (eg, a vehicle, a pedestrian) without driver intervention, and drive to a set destination based on the recognition. Autonomous vehicles are classified from level 1 to level 4 according to the degree of driver intervention, of which level 4 can support a fully autonomous driving state in which the system performs all operations including acceleration, steering and braking. .

Korean published application 10-2018-0125885

Autonomous vehicles require vehicle control techniques that can prepare for various situations that may be caused by surrounding vehicles and surrounding objects under road regulations to prevent accidents. For example, autonomous vehicles need to quickly recognize hazards on the road and avoid the recognized hazards.

Various embodiments disclosed in this document provide a camera-based road event occupied lane detection apparatus and method capable of detecting a lane occupied by a road event occurring on a road around the vehicle by analyzing an image captured by a camera. can

According to the present invention, there is provided an apparatus for detecting a road event occupied lane based on a camera, comprising: a camera for generating an image by photographing a road around a vehicle; a memory storing at least one instruction; and a processor, wherein, as the at least one instruction is executed, an image obtained by photographing a surrounding road of the vehicle from the camera is obtained, and a road event acting as a risk factor on the surrounding road from the image; When a lane is recognized and the road event is recognized, the angle between the reference line and the road event and the angle between the reference line and the lane are compared in the image to determine the road event occupied lane occupied by the road event, wherein the reference line is the lane It may be a line above the vanishing point in a vertical line of the image passing through the vanishing point.

The processor may recognize a lane in the image, and detect the total number of lanes and a driving lane from the recognized lane.

The processor is configured to detect a first edge line connecting the vanishing point and a right edge of the road event and a second edge line connecting the vanishing point and a left edge of the road event, and the first and second edge lines and the The road event occupation lane may be determined by detecting a road event occupancy angle that is an angle between reference lines, detecting a lane angle between the lane and the reference line, and comparing the road event occupation angle with the lane angle.

The road event may include any one of a falling object falling on the road around the road, a hole occurring on the road, an accident vehicle, and a rubber cone indicating a construction area or an accident area.

The memory stores data related to the object, the processor detects the object from the image based on the data, and for a designated type of object among the detected objects, among a movement speed, a movement direction, and a size At least one variable may be checked, and it may be determined whether the road event corresponds to the at least one variable based on the at least one variable.

The road event occupied lane detecting device may further include an output device, and the processor may output an image or sound related to the road event occupied lane through the output device.

The road event occupied lane detecting apparatus may further include a communication circuit, and the processor may transmit or receive information about the road event occupied lane to the outside using the communication circuit.

In addition, the method for detecting a road event occupied lane based on a camera according to the present invention comprises the steps of: (a) acquiring an image of a road around the vehicle from a camera; (b) recognizing a road event and lane acting as a risk factor on a surrounding road from the image; and (c) when the road event is recognized, comparing the angle between the reference line and the road event and the angle between the reference line and the lane in the image to determine the road event occupied lane occupied by the road event, wherein the reference line may be a line above the vanishing point in a vertical line of the image passing through the vanishing point of the lane.

According to various embodiments disclosed in this document, it is possible to easily detect a lane occupied by a road event occurring on a road around a vehicle using only an image captured by a camera without the aid of a Global Navigation Satellite System (GNSS) or map information. , can help a lot in autonomous driving of vehicles. In addition, various effects directly or indirectly identified through this document may be provided.

1 is a diagram for explaining a camera-based road event occupied lane detection method according to the present invention.
2 is a block diagram of a camera-based road event occupied lane detection apparatus according to the present invention.
3 is a schematic flowchart of a camera-based road event occupied lane detection method according to the present invention.
4 is a detailed flowchart of a camera-based road event occupied lane detection method according to the present invention.
5 illustrates a method of detecting a road event occupied lane in a left direction of a vehicle according to an exemplary embodiment.
6 illustrates a method for detecting a road event occupied lane in a right direction of a vehicle according to an exemplary embodiment.
In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

1 is a diagram for explaining a camera-based road event occupied lane detection method according to the present invention.

Referring to FIG. 1 , the road event occupied lane detecting apparatus 110 may acquire an image (still image or video) of at least one driving direction of at least one vehicle using a camera. The camera may photograph, for example, at least one of the front, rear, and left and right directions of the vehicle.

The road event occupied lane detection device 110 recognizes a traffic line (traffic lane) by applying a computer vision image processing technique to an image captured by a camera, and detects a traffic path based on the lane recognition result can do. Furthermore, the road event occupied lane detecting apparatus 110 may acquire the total number of lanes and driving lane information, which is the lane in which the vehicle is traveling, by using information on the detected lane.

According to the present invention, the road event occupancy lane detection device 110 is a Hough Transform method, an Inverse Perspective Mapping (IPM) method, a RANSAC line Fitting (RANdom SAmple Consensus) method, The lane may be recognized by applying at least one algorithm of a Beizer Spline Fitting method. However, the present invention is not limited thereto, and lanes may be recognized using various other methods or algorithms.

The road event occupied lane detection device 110 analyzes an object included in the acquired image according to an unsupervised learning-based artificial intelligence technology, and based on the analyzed object, a road event with a risk of an accident on the surrounding road It can be checked whether 120 exists. The road event occupied lane detecting apparatus 110 may detect a lane and a lane by using an image obtained from a camera mounted on the front and/or rear of the vehicle and recognize the road event. Of course, the road event occupied lane detecting device 110 can detect lanes and lanes using images acquired from cameras mounted on the left and/or right and recognize a road event, as well as being mounted on at least one of front, rear, left, and right. Of course, lanes and lanes can be detected and road events can be recognized by using the image obtained from the camera.

The unsupervised learning may be a method of analyzing a desired object (eg, an object) by learning inference according to a relationship between learning data when there is no target value for a learning result. The road may include, for example, a road on which a vehicle including at least one traffic path can travel. A portion of a road divided into traffic lanes (traffic lanes) on a road on which a vehicle travels is called a traffic path. The lane may be, for example, a road divided by at least one of a center line (eg, a solid yellow line) and a lane (eg, a dotted white line).

On the other hand, in the present invention, the road event 120 means an object that can appear on the road except for the vehicle being driven, for example, a falling object on the road, a hole (eg, a sink) generated on the road. It may include any one of a sinkhole, a porthole, an accident vehicle, a construction area, or a rubber cone marking the accident area.

A vanishing point is a point where parallel straight lines in physical space are projected onto an image and appear to meet at one point by a perspective effect. And, the vanishing line is a straight line formed by gathering vanishing points. Vanishing points of straight lines in a parallel plane form a straight line. Also, a vanishing line for a plane is an intersection that is formed when a plane that is horizontal to this plane and passes through the camera origin meets the image plane.

That is, if a straight line on the ground is projected onto an image, it can be easily imagined that if the straight line is extended infinitely, it will eventually converge to a point on the vanishing line.

The position of the vanishing line in the image is independent of the camera's translation and is affected only by the camera's posture (3D rotation). Therefore, if the position of the vanishing line in the captured image taken with the camera is known, the pan, tilt, and roll of the camera can be determined without calibration. That is, it is possible to measure the 3D posture of the camera using the vanishing point/vanishing line. However, by using the vanishing point/vanishing line in the image, it is possible to estimate not only these external parameters but also the internal parameters of the camera. That is, it is possible to estimate the focal length and main point of the camera.

As a result, the road event occupied lane detection device 110 recognizes a lane in a captured image captured by a camera mounted on a vehicle, detects a lane from the recognized lane, and because the lanes are parallel on the road, the lanes gather as a vanishing point. can be used to detect the vanishing point of the lane.

When the road event occupied lane detection device 110 identifies the road event 120 on the surrounding road, the two roads between the vehicle and the road event 120 from the image obtained based on the vanishing point (eg, the vanishing point as the centripetal point) You can check the event occupancy angle and the lane angle for all lanes. The lane may include, for example, a lane dividing each lane and a center line.

Here, the road event occupancy angle includes a first occupancy angle and a second occupancy angle.

If the upper line in the vertical line of the image passing through the vanishing point of the lane is the reference line (A), the first occupancy angle (β1) is the first edge line (B1) and the reference line (A) connecting the vanishing point and the right edge of the road event. and the second occupancy angle β2 is an angle between the second edge line B2 connecting the vanishing point and the left edge of the road event and the reference line A.

And, the lane angle is the angle (α1, α2, α3) formed with all the lanes and the reference line A. There may be a left lane and a right lane based on the driving lane, and the angle formed by the respective lanes with the reference line in order from the rightmost lane becomes the lane angle.

The road event occupied lane detecting apparatus 110 may determine a lane occupied by the road event 120 by comparing two road event occupancy angles and lane angles with each other.

For example, as shown in FIG. 1 , the vehicle is traveling in the forward direction from the two-lane road to the first lane, and the road event 120 may exist in the right direction of the vehicle. In this case, the road event occupied lane detecting device 110 connects the right and left edges of the vanishing point and the road event 120 to the reference line A, which is a line at the top of the vanishing point in the vertical line of the image passing the vanishing point of the lane. B1, B2) may be determined, and the occupancy angles (β1, β2) of the road event between the determined baseline and the two edge lines may be confirmed, respectively.

In the case of FIG. 1 , the road event occupied lane detecting device 110 determines a vanishing point and a reference line A from the acquired image, and a first straight line (hereinafter referred to as a 'comparison line') connecting the vanishing point and the rightmost lane. ), a second straight line connecting the vanishing point and a lane on the left side of the first straight line, and a third straight line connecting the vanishing point and a lane on the left side of the second straight line are determined, and the second line between the determined reference line A and the first straight line is determined. The first lane angle α1, the second lane angle α2 between the reference line A and the second straight line, and the second lane angle α3 between the reference line A and the third straight line may be identified, respectively. Since the road may consist of a plurality of lanes, the first to nth lane angles may be sequentially provided from the rightmost lane in the lane angle between the reference line and the lane.

Since the road event 120 may exist on the left or right side in the driving direction of the vehicle, and there may be a plurality of road events, the road event occupancy lane detecting device 110 independently identifies each event and at the same time determines the road for each road event. Both the event occupancy angle and the lane angle can be determined.

The road event occupied lane detecting device 110 compares the road event occupancy angles β1 and β2, the first lane angle α1, the second lane angle α2, and the third lane angle α3 with each other to determine a road event. (120) may determine the occupied lane. For example, since the road event occupancy angles β1 and β2 corresponding to the road event are greater than the first lane angle α1 corresponding to each lane and smaller than the second lane angle α2, the road event occupied lane detection device 110 may determine a lane occupied by the road event 120 as a second lane.

Of course, the road event may be within one lane, or may occupy all of several lanes. In FIG. 1, when the first occupancy angle β1 among the road event occupancy angles is smaller than the second lane angle α2 and the second occupancy angle β2 is greater than the second lane angle α2, the road event 120 may be determined to exist over the driving lane and the right lane thereof.

When the road event occupied lane detecting device 110 determines the lane occupied by the road event 120, to prevent a vehicle accident due to the road event, the device 110 controls or supports the vehicle to bypass the lane occupied by the road event 120 can Additionally or alternatively, the road event occupied lane detecting device 110 may display the location of the road event 120 (eg, a lane occupied by the road event) through an output device (eg, at least one of a display or a speaker). can guide you

According to an embodiment, the road event occupancy lane detection device 110 includes an in-vehicle head unit, an embedded board, a smartphone, a tablet PC, a PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a vehicle, and a media. It may be, but is not limited to, a player, a micro server, a Global Navigation Satellite System (GGNSS) device, an e-book terminal, a digital broadcast terminal, a navigation device, a kiosk, an MP3 player, a digital camera, a home appliance, and other mobile or non-mobile computing devices. . For example, the road event occupancy lane detection device 110 obtains an image (eg, a video and a still image) from a camera, and provides a notification message to the user based on the obtained image It can include all kinds of devices that can.

The road event occupied lane detection device 110 is a module installed in the vehicle, can control the operation of the vehicle, communicate with other modules installed in the vehicle through a predetermined network, and exchange information on the road event occupied lane. . The road event occupied lane detecting device 110 may be a device separate from the vehicle, such as a smartphone, and an image may be obtained by using the camera of the road event occupied lane detecting device 110 or a camera of another device installed in the vehicle. Also, an image may be received from a camera capable of photographing the surroundings of the vehicle through a designated network. The network includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and combinations thereof. may include A vehicle may be a means of transportation, such as a car, bus, truck, train, bicycle, motorcycle, etc., having a communication function, a data processing function, and a transportation function. The vehicle may be a driving simulator system capable of detecting a vehicle road event using virtual reality and a real-time vehicle analysis program. The simulator produces a real or virtual dynamic system model and then tests and evaluates it through a computer program, and can be used to efficiently secure driving data according to the movement of a vehicle in virtual reality.

According to the above-described embodiment, the road event occupied lane detecting device 110 is based on the road event occupancy angle based on the vanishing point identified in the image taken around the vehicle and the angle between the lanes, the road event with the risk of an accident is occupied One lane can be detected quickly. In addition, the road event occupied lane detecting device 110 may detect the lane occupied by the road event using only the angle with respect to the location direction (eg, left or right) of the road event based on the vehicle, so that the road event location is detected. It is possible to reduce the number of operations and increase the processing speed.

2 is a block diagram of a camera-based road event occupied lane detection apparatus according to the present invention.

Referring to FIG. 2 , the camera-based road event occupied lane detecting apparatus 110 according to the present invention may include a camera 111 , a memory 113 , and a processor 119 . In an embodiment, the road event occupied lane detecting apparatus 110 may omit some components or further include additional components. For example, the road event occupied lane detecting device 110 may further include at least one of a communication circuit 115 and an output device 117 .

For another example, the road event occupancy lane detecting device 110 does not include the camera 111 , but an image from another device (eg, an around view system) or a camera 111 provided in an aftermarket product. can be obtained (eg, received). In addition, some of the components of the road event occupancy lane detecting device 110 are combined to form a single entity, and the functions of the components prior to the combination may be performed identically.

The camera 111 may capture a still image or a moving image according to a command of the processor 119 , and may output the captured still image or moving image. The camera 111 may be implemented based on, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. The camera 111 may be provided to photograph at least one of the front, rear, and left and right sides of the vehicle.

The memory 113 may store, for example, a command or data related to at least one other component of the road event occupancy lane detection device 110 . The memory 113 may be a volatile memory (eg, RAM, etc.), a non-volatile memory (eg, ROM, flash memory, etc.), or a combination thereof. The memory 113 may store at least one instruction that, when executed, causes the processor 119 to detect a road event. The at least one instruction may be, for example, capable of recognizing and detecting an object included in the image, confirming the type of the detected object, and identifying at least one variable among a moving speed, a moving direction, and a size for a designated object. can The memory 113 may store object-related data and/or road event-related data for detecting types of objects and road events. The types of objects are, for example, driveways, sidewalks, buildings, walls, guardrails, poles, traffic lights, traffic signs, props, plants, sky, people, automobiles, trucks, buses, trains, motorcycles, bicycles and safety tripods. It may include at least one, and the type of road event includes a falling object that fell on the road, a hole that occurred on the road (eg, a sinkhole, a pothole), an accident vehicle, and a rubber cone indicating a construction area or an accident area. ) may include at least one of.

The communication circuit 115 may support establishment of a communication channel or wireless communication channel between the road event occupied lane detecting device 110 and other devices (eg, other devices in the vehicle), and performing communication through the established communication channel. The communication channel may be a communication channel including a vehicle network, for example, a controller area network (CAN) and a media oriented systems transport (MOST). The communication channel includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and combinations thereof. It may include a communication channel according to The communication circuit 115 may exchange information on the road event occupied lane obtained according to the present invention with an external device, and a GNSS module that receives a GNSS (Global Navigation Satellite System) signal and calculates the vehicle's current location. may include

The output device 117 may include at least one of a speaker capable of outputting a sound or a display capable of outputting an image according to a command of the processor 119, and may output information about a road event occupied lane as an image or sound. .

The processor 119 may use the instructions stored in the memory 113 to execute an operation or data processing related to control and/or communication of at least one other component of the road event occupancy lane detecting apparatus 110 . The processor 119 may include, for example, a central processing unit (CPU), a graphic processing unit (GPU), a microprocessor, an application processor, an application specific integrated circuit (ASIC), or field programmable gate arrays (FPGA). )), and may have a plurality of cores.

The processor 119 acquires an image (eg, a front image) including a road around the vehicle by using the camera 111 , and extracts a region of interest (ROI) expected as a road region from the acquired image. can do.

The processor 119 may recognize a lane by applying a computer vision image processing technique to an image captured by a camera, and detect a vanishing point and a traffic path based on the lane recognition result.

In addition, the processor 119 may detect the total number of lanes and driving lane information on the driving road. Of course, the total number of lanes and driving lane information may be directly recognized and detected by the processor 119, or the total number of lanes and driving lane information may be acquired from other external devices or modules.

The processor 119 may analyze and classify an object detected in an image according to an unsupervised learning-based artificial intelligence technology to detect a road event existing on a road around the vehicle. For example, the processor 119 classifies the type of object on the road using a machine learning or deep learning analysis module, and thinks the rest except for the object that can classify the type of object (eg, vehicle, lane, road sign). It can be detected as a dangerous road event. As another example, if the processor 119 determines that an unknown type object exists on a surrounding road based on the object-related data, the processor 119 may determine the unknown object as a road event with an accident risk.

The machine learning or deep learning analysis module is a module for classifying objects on the road learned through unsupervised learning according to object-related data based on a clustering algorithm, and is a software module implemented by the processor 119. can The unsupervised learning may be, for example, a method of analyzing or classifying a desired object (eg, an object) by learning inference according to a relationship between learning data when there is no target value for a learning result. The unsupervised learning may be effective in classifying an object (eg, a falling object on a road, a hole, an accident point, a danger point) for which it is difficult to construct the learning data for the learning guidance because the form or shape is not determined.

The processor 119 may exclude an object in an exceptional situation from among the detected road events. The object in the exceptional situation may include, for example, a person object located on a crosswalk.

The processor 119 may recognize a lane (eg, a dotted line and a center line) of a road based on the type of the classified object. For example, when recognizing an object corresponding to a pattern of a lane according to object-related data in an image, the processor 119 may determine the recognized object as a lane.

Additionally or alternatively, the processor 119 may recognize a lane based on information on the total number of lanes and/or an object detected from the image. The total number of lanes information may be, for example, lane information (eg, may be included in map information) stored in the memory 113 corresponding to the current location of the vehicle identified through the communication circuit 115 . For example, the processor 119 determines a road area based on a distance between an object (eg, a guard rail, a building, a tree) detected from the image and a vehicle, and determines a road area from the image based on information on the road area and the total number of lanes. width can be estimated. The processor 119 may recognize a lane existing in a road around the vehicle based on the estimated lane width. The processor 119 may recognize a surrounding road based on the recognized road area or the recognized lane. The surrounding road may include a lane in which the vehicle is traveling and at least one adjacent lane.

When the processor 119 checks the road event existing on the surrounding road, in order to determine the lane occupied by the road event, based on the vanishing point of the image (for example, with the vanishing point as the central point) between the vehicle and the road event 120 . It is possible to determine the occupancy angles of two road events and the lane angles for all lanes.

The processor 119 detects an edge in the image acquired from the camera 111 using, for example, an edge detection algorithm (eg, a canny edge detector), and uses, for example, a Hough transform. Thus, a point at which the linear components gather the most at the detected edge may be determined as the vanishing point of the image. The Hough transform may be to convert pixels into a straight line in an image from which noise other than an edge has been removed.

For example, the processor 119 may check the reference line A and the road event occupancy angle. Here, the road event occupancy angle includes a first occupancy angle and a second occupancy angle, and the first occupancy angle β1 is between the first edge line B1 connecting the vanishing point and the right edge of the road event and the reference line A. angle, and the second occupancy angle β2 is an angle between the second edge line B2 connecting the vanishing point and the left edge of the road event and the reference line A.

The processor 119 may determine the lane occupied by the road event on the road by comparing the occupancy angles of the two road events and the lane angles, respectively.

As such, when the occupancy angles of the two road events are compared with the lane angles according to each lane, the processor 119 may determine the lane occupied by the road event. Of course, since the road event is large, several lanes may be occupied together, and the processor 119 may determine even such a case.

In the above-described embodiment, the processor 119 may confirm that the road event exists on the lane of the vehicle being driven. In this case, the processor 119 may determine the lane in which the vehicle is driving as the lane occupied by the road event.

When the processor 119 determines the lane occupied by the road event, the processor 119 may control the driving of the vehicle to bypass the road event. The processor 119 may output information (eg, image or sound) related to the road event occupied lane through the output device 117 . For example, the processor 119 may output a sound "There is a road event with a risk of an accident ahead" through a speaker. Of course, information on the direction of the road event and the occupied lane can also be output.

According to various embodiments, the processor 119 may detect a road event existing on a road around the vehicle based on at least one of a moving speed, a moving direction, and a size of the detected object. For example, the processor 119 may acquire a plurality of consecutive images from the camera 111 and determine whether a road event exists on the road based on a change in the position of an object detected from the plurality of images. The processor 119 may check, for example, a change in the position of the object in the plurality of images, and determine at least one of the moving speed, the moving direction, and the size of the object based on the current speed of the vehicle and the confirmed position change of the object. can

The processor 119 may estimate the size of the object based on, for example, a lane width of a road. The processor 119 may determine the moving speed of the object, for example, based on a change in the position of the object included in the image. The processor 119 may, for example, determine the size of the object included in the image, and determine the distance to the object according to the size of the object based on size/distance related data (stored in the memory 113 ). . According to the above-described embodiment, the processor 119 detects the recognized object as a road event if there is an abnormality in the change of the corresponding object (eg, an object related to a vehicle stopped on the road) even if the object has recognized the type of the object. can do.

According to various embodiments, at least some of the lanes recognized based on the image (eg, an area close to the vanishing point) may be curved lanes. In this case, the processor 119 converts the lane in the direction corresponding to the road event (eg, the direction of the road event with respect to the reference line) among the lanes recognized based on the vanishing point into a straight lane, and converts the lane based on the converted straight lane into a straight lane. The lane in which the event is located can be detected.

According to various embodiments, when the processor 119 receives road event information from another device (eg, a traffic information providing server) through the communication circuit 115 , and confirms a road event detected from the received road event information, may not check the lane occupied by the road event. If it is possible to confirm the detected road event from the received road event information, the processor 119 determines the lane occupied by the road event as described above and provides information on the lane occupied by the road event to another device through the communication circuit 115 . can be sent.

In the above-described embodiment, it has been described as an example that the road event occupied lane detecting device 110 detects a road event on the driving road based on the image of the camera 111 during autonomous driving. However, the present invention is not limited thereto. For example, the road event occupied lane detecting apparatus 110 may detect a road event on the driving road based on an image obtained using the camera 111 when providing a route guidance function through navigation.

According to the above-described embodiment, the road event occupied lane detecting device 110 can quickly detect the lane occupied by the road event based on the angle between the road event and the lane based on the vanishing point identified in the image taken around the vehicle. have. In addition, the road event occupied lane detecting apparatus 110 may detect a lane occupied by the road event by using the direction of the road event with respect to the vehicle and the road event occupancy angle, thereby reducing the amount of calculation and increasing the processing speed.

3 is a schematic flowchart of a method for detecting a road event occupied lane according to the present invention.

The method for detecting a road event occupied lane according to the present invention is a method performed by the above-described device for detecting a road event occupied lane according to the present invention, and since it is substantially the same invention, the overlapping description will be omitted.

Referring to FIG. 3 , in S100 , the road event occupied lane detecting device (eg, the road event occupied lane detecting device 110 of FIG. 2 ) is a peripheral road photographed by a camera (eg, the camera 111 of FIG. 2 ). It is possible to obtain an image including For example, the road event occupied lane detecting apparatus 110 may acquire an image including a surrounding road in the vehicle driving direction from the camera 111 . As another example, the road event occupied lane detecting apparatus 110 may acquire an image including a road around the vehicle from the camera 111 mounted on at least one of the front, rear, and side of the vehicle.

In S200 , the road event occupied lane detecting apparatus 110 may recognize a road event from the acquired image to determine whether a road event with a risk of an accident exists on a road around the vehicle. For example, the road event occupied lane detecting apparatus 110 may check whether a road event satisfying a specified condition exists in the acquired image. The specified condition is, for example, at least one of a case in which the type of an object existing on the surrounding road cannot be confirmed and a case in which an object existing on the surrounding road is larger than a specified size and less than a specified speed and is moving in a direction toward a vehicle may include.

In S300 , when a road event is recognized, the road event occupied lane detecting apparatus 110 may compare two road event occupancy angles with a lane angle to determine a road event occupied lane on the road.

For example, the road event occupied lane detecting apparatus 110 may identify the road event occupancy angle including the first occupancy angle and the second occupancy angle based on the vanishing point. The first occupancy angle β1 is the angle between the first edge line B1 and the reference line A connecting the vanishing point and the right edge of the road event, and the second occupancy angle β2 is the vanishing point and the left edge of the road event. It is an angle between the connected second edge line (B2) and the reference line (A).

The road event occupied lane detecting apparatus 110 may check a direction in which a road event is located with respect to the reference line, and may check lane angles between all lanes existing in the identified direction based on the vanishing point and the reference line. The processor 119 may compare two road event occupancy angles with each lane angle to determine a lane occupied by the road event on the road.

4 is a detailed flowchart of a method for detecting a road event occupied lane according to an embodiment.

Referring to FIG. 4 , in S400 , the road event occupied lane detecting device (eg, the road event occupied lane detecting device 110 of FIG. 2 ) includes the surrounding road from the camera (eg, the camera 111 of FIG. 2 ). image can be obtained. For example, the road event occupied lane detecting apparatus 110 may acquire an image including a surrounding road in the vehicle driving direction from the camera 111 . As another example, the road event occupied lane detecting apparatus 110 may acquire an image including a road around the vehicle from the camera 111 mounted on at least one of the front, rear, and side of the vehicle.

In S410 , the road event occupied lane detecting apparatus 110 may recognize a lane from the image and detect the lane to obtain the total number of lanes and information on the driving lane in which the vehicle is traveling.

In S420, the road event occupied lane detecting apparatus 110 determines a vanishing point and a reference line from the acquired image, and recognizes the road event. In this case, the road event occupied lane detecting apparatus 110 may analyze the acquired image to determine whether a road event that may act as a risk factor exists on a road around the vehicle.

For example, the road event occupied lane detecting apparatus 110 may determine whether a road event satisfying a specified condition exists on a road around the vehicle detected from the acquired image. The specified condition is, for example, at least one of a case in which the type of an object existing on the surrounding road cannot be confirmed and a case in which an object existing on the surrounding road is larger than a specified size and less than a specified speed and is moving in a direction toward a vehicle may include.

Although step S420 is shown to be performed after step S410 in FIG. 4 , steps S410 and S420 may be performed simultaneously or the order of each other may be changed, and may also be performed independently of each other.

Then, the road event occupied lane detecting apparatus 110 may detect two road event occupancy angles between the reference line and the road event from the image in S430 .

In S440 , the road event occupied lane detecting apparatus 110 may detect a lane angle between all lanes and a reference line.

In S450, the road event occupied lane detecting apparatus 110 may compare two road event occupancy angles with each lane angle to determine a road event occupied lane that is a lane occupied by a road event on the road.

As such, by comparing two road event occupancy angles with each lane angle according to each lane, the road event occupied lane detecting apparatus 110 may determine a lane occupied by the road event. Of course, since the road event is large, several lanes may be occupied together, and the road event occupied lane detecting apparatus 110 may determine even such a case.

5 illustrates a method for detecting a road event occupied lane on a road in a left direction of a vehicle according to an exemplary embodiment.

Referring to FIG. 5 , the vehicle is traveling on the third lane among the 5-lane roads, and the road event 120 (eg, the road event 120 of FIG. 1 ) may exist in the left direction of the vehicle 110 .

In this case, the road event occupied lane detecting device 110 connects the vanishing point 0 to the right edge of the road event 120 in the obtained image, and the first edge line B1 and the vanishing point 0 and the road event 120 are connected. ), a second edge line B2 connecting the left edge of ), the second occupancy angle β2 can be confirmed.

In addition, the road event occupied lane detecting device 110 checks the first lane angle α1 between the reference line A and the right lane of the rightmost lane based on the reference line, and sequentially clockwise angles of the second lane ( α2), the third lane angle α3, the fourth lane angle α4, the fifth lane angle α5, and the sixth lane angle α6 may be checked, respectively.

If the road event occupancy angles β1 and β2 are less than or equal to the sixth lane angle α6 and greater than the fifth lane angle α5, the road event occupied lane detecting device 110 sets the lane occupied by the road event 120 to 1 It can be decided by car. If the road event occupancy angles β1 and β2 are less than or equal to the fifth lane angle α5 and greater than the fourth lane angle α4, the road event occupied lane detecting device 110 sets the lane occupied by the road event 120 to 2 It can be decided by car.

Of course, a road event may exist across at least one lane, and the road event occupied lane detecting device 110 compares two road event occupancy angles with each lane angle to determine both the direction of the road event and the occupied lane. can do.

In the case of FIG. 5, since the road event occupancy angles β1 and β2 are both less than the fifth lane angle α5 and exceed the fourth lane angle α4, the road event occupied lane detecting device 110 detects the road event 120 ) can be determined as a two-lane lane.

According to the above-described embodiment, the road event occupied lane detecting apparatus 110 may determine the lane occupied by the road event 120 by using a lane corresponding to the direction in which the road event 120 is located, so that the road event location is detected. It is possible to reduce processing load and calculation due to unnecessary analysis of a lane in a direction in which the road event 120 is not located.

6 illustrates a method for detecting a road event occupied lane on a road in a right direction of a vehicle according to an exemplary embodiment.

Referring to FIG. 6 , the vehicle is traveling on a five-lane road, and the road event 120 (eg, the road event 120 of FIG. 1 ) may exist in the right direction of the vehicle 110 .

In this case, the road event occupied lane detecting device 110 connects the vanishing point 0 to the right edge of the road event 120 in the obtained image, and the first edge line B1 and the vanishing point 0 and the road event 120 are connected. ), a second edge line B2 connecting the left edge of The second occupancy angle β2, which is an angle between the two, can be confirmed.

In addition, the road event occupied lane detecting device 110 checks the first lane angle α1 between the reference line A and the right lane of the rightmost lane based on the reference line, and sequentially clockwise angles of the second lane ( α2), the third lane angle α3, the fourth lane angle α4, the fifth lane angle α5, and the sixth lane angle α6 may be checked, respectively.

The first occupancy angle β1 of the road event exceeds the first lane angle α1 and is less than the second lane angle α2, and the second occupancy angle β2 of the road event exceeds the second lane angle α2 Since it exceeds and is less than the third lane angle α3, the road event occupied lane detecting apparatus 110 may determine that the road event 120 occupies the fourth and fifth lanes together.

If the road event occupancy angles β1 and β2 are less than or equal to the sixth lane angle α6 and greater than the fifth lane angle α5, the road event occupied lane detecting device 110 determines the lane occupied by the road event 120. You will be able to decide with 5 lanes.

As shown in FIG. 6 , a road event may exist over at least one lane, and the road event occupied lane detection device 110 compares two road event occupancy angles with each lane angle to determine the direction of the road event and the occupied lane. all can be judged.

It should be understood that the various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "A; Each of the phrases "at least one of B, or C" may include any one of, or all possible combinations of, items listed together in the corresponding one of the phrases. Terms such as “first”, “second”, or “first” or “second” may simply be used to distinguish the component from other components in question, and may refer to components in other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively". When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

As used herein, the term “module” may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, component, or circuit. A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are stored in a storage medium (eg, internal memory 1436 or external memory 1438) readable by a machine (eg, road event occupied lane detection device 1401). It may be implemented as software (eg, program 1440) including one or more stored instructions. For example, the processor (eg, the processor 1420) of the device (eg, the road event occupancy lane detecting device 1401) may call at least one of the one or more instructions stored from the storage medium and execute it. have. This makes it possible for the device to be operated to perform at least one function according to the at least one command called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided in a computer program product (computer program product). Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly, online between smartphones (eg: smartphones). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily created in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, a module or a program) of the above-described components may include a singular or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, or omitted. or one or more other operations may be added.

110: road event occupied lane detection device
111: camera
113: memory
115: communication circuit
117: output device
119: processor

Claims (14)

A camera for generating an image by photographing the surrounding road of the vehicle;
a memory storing at least one instruction; and
including a processor;
As the processor executes the at least one instruction,
Obtaining an image of the surrounding road of the vehicle from the camera,
Recognizing a road event and lane acting as a risk factor on the surrounding road from the image, the road event is recognized by machine learning or deep learning,
When the road event is recognized, the road event occupied lane occupied by the road event is determined by comparing the angle between the reference line and the road event in the image and the angle between the reference line and the recognized lane,
The reference line is a line at the top of the vanishing point in the vertical line of the image passing through the vanishing point of the lane,
The processor is
A first edge line connecting the vanishing point and the right edge of the road event and a second edge line connecting the vanishing point and the left edge of the road event are detected, and the angle between the first and second edge lines and the reference line is Detecting a road event occupancy angle, detecting a lane angle between the reference line and the recognized lane, comparing the road event occupancy angle with the lane angle, respectively, to determine the road event occupied lane, A camera-based road event occupied lane detection device, characterized in that it detects the number of lanes and driving lanes. delete delete The method according to claim 1, The road event,
A camera-based road event occupancy lane detection device including any one of a falling object falling on the road on the surrounding road, a hole occurring on the road, an accident vehicle, and a rubber cone indicating a construction area or an accident area . The method according to claim 1,
The memory stores data related to the object,
The processor is
detecting the object from the image based on the data;
Checking at least one variable among a moving speed, a moving direction, and a size for a specified type of object among the detected objects,
A camera-based road event occupancy lane detection apparatus for checking whether the road event corresponds to the road event based on the at least one variable. The method according to claim 1, wherein the device,
further comprising an output device;
The processor is configured to output an image or sound related to the road event occupied lane through the output device, a camera-based road event occupied lane detection device. The method according to claim 1, wherein the device,
further comprising a communication circuit;
wherein the processor transmits or receives information about the road event occupied lane to the outside using the communication circuit or receives from the outside, a camera-based road event occupied lane detecting device. (a) obtaining an image of the vehicle surrounding the road from the camera;
(b) recognizing a road event and lane acting as a risk factor on the surrounding road from the image;
(c) detecting the total number of lanes and driving lanes from the recognized lanes; and
(d) when the road event is recognized, determining the road event occupied lane occupied by the road event by comparing the angle between the reference line and the road event in the image and the angle between the reference line and the recognized lane,
The reference line is a line at the top of the vanishing point in the vertical line of the image passing the vanishing point of the lane, and the road event is recognized by machine learning or deep learning,
Step (d) is,
detecting a first edge line connecting the vanishing point and a right edge of the road event and a second edge line connecting the vanishing point and a left edge of the road event;
detecting a road event occupancy angle that is an angle between the first and second edge lines and the reference line;
detecting a lane angle between the reference line and the recognized lane; and
and determining the road event occupied lane by comparing the road event occupancy angle with the lane angle, respectively. delete delete The method according to claim 8, The road event,
A camera-based road event occupancy detection method, including any one of a falling object falling on the road on the surrounding road, a hole occurring on the road, an accident vehicle, and a rubber cone indicating a construction area or an accident area . The method according to claim 8, wherein the step (b),
detecting the object from the image based on data related to the object stored in a memory;
checking at least one variable among a moving speed, a moving direction, and a size with respect to a specified type of object among the detected objects; and
and recognizing a road event by checking whether the road event corresponds to the road event based on the at least one variable. The method of claim 8, wherein the method comprises:
The method of claim 1, further comprising outputting an image or sound related to the road event occupied lane using an output device. The method of claim 8, wherein the method comprises:
The method of detecting a road event occupied lane based on a camera further comprising the step of transmitting or receiving information about the occupied lane to the outside using a communication circuit.

Images