WO2023101221A1

WO2023101221A1 - Driving scenario recommendation method and device

Info

Publication number: WO2023101221A1
Application number: PCT/KR2022/016468
Authority: WO
Inventors: 박종빈; 김경원
Original assignee: 한국전자기술연구원
Priority date: 2021-12-02
Filing date: 2022-10-26
Publication date: 2023-06-08
Also published as: KR20230082777A

Abstract

The present invention relates to a driving scenario recommendation method and device. The method, according to one embodiment of the present invention, is carried out on an electronic device in order to recommend driving scenarios of a vehicle, the method comprising: an analysis step for, from a driving image including photographing location information, analyzing driving statistics including the number of objects, object movement trajectories, and driving events, according to the location; a first recommendation step for, on the basis of the driving statistics, recommending, for a road needing scenarios to be set therefor, scenario sections and driving scenarios applying to the respective scenario sections; and a second recommendation step for, on the basis of the driving statistics, recommending parameters suitable for the recommended driving scenarios for the respective scenario sections.

Description

Driving scenario recommendation method and device

The present invention relates to a method and apparatus for recommending driving scenarios, and more particularly, to a method for accumulating and analyzing events that occur during driving based on driving images, and recommending an appropriate driving scenario for a target road based on the analyzed information, and It's about the device.

In order to develop a human-driven vehicle or an autonomous vehicle that minimizes human control, various functional evaluations, performance evaluations, and safety evaluations are required. Conventionally, for this evaluation, an overall scenario for vehicle driving is prepared in advance, and various tests are performed according to the prepared scenario. For example, an integrated scenario is created by combining partial event scenarios such as "cutting in", "lane reduction", "vehicle in front accident", "merging roads", "appearance of pedestrian", etc. By applying, it was tested and evaluated whether the existing vehicle or the vehicle under development responds appropriately and has no problems.

To create this scenario, a "top-down approach" has traditionally been applied. In other words, it has been common for the subject who creates the scenario to select an arbitrary event for the road section to be tested, set parameters such as the number of vehicles driving and road congestion, configure the integrated scenario, and then perform the simulation.

That is, a conventional process for creating a driving scenario is summarized as follows.

(1) The subject who creates the scenario divides the random road section that needs testing into the partial road section to which the partial scenario is to be applied.

(2) Configure scenarios that combine appropriate incident situations for each section of the road.

(3) Parameters such as the number of vehicles driving or road congestion are determined for each sub-scenario.

(4) In the final simulation or actual road test stage, a computer-based simulation or actual road test is conducted using scenarios and parameters set for each section.

In the case of this conventional method, the degree of freedom of testing is high in terms of being able to test various possibilities. However, when the conventional method is applied, it takes a lot of time and resources to test it as too many cases are generated, and there is a problem that test scenarios and parameters that are far from reality can be applied. For example, a scenario that rarely occurs at a specific road location may be generated and applied to the test, and problems such as setting an excessive number of vehicles may occur. Of course, scenarios that include extremely rare and exceptionally occurring events can also be meaningful, but if you know how often the scenarios occur and configure them, trial and error can be reduced and scenarios that match the real situation can be constructed.

In addition, in acquiring statistical information generated on the road, conventionally, the focus has been on collecting and managing information that can be acquired after a specific accident occurs. That is, while driving, such as "frequency of cut-in", "frequency of sudden stop", "overtaking frequency", etc., even if no actual accident occurs, driving events that may cause congestion or potential accidents in a certain road section There is no technology to collect and manage information about

That is, the main problems or limitations of the prior art in relation to generation of driving scenarios are summarized as follows.

(1) In a given road environment, an objective method is needed to determine the detailed section to which the driving scenario is applied. For example, statistics on driving events are needed to analyze potential accident risk factors and road congestion factors that occur in everyday driving situations.

(2) An objective plan is needed regarding which scenario to apply in a given road environment.

(3) It is necessary to design scenarios related to actual road conditions.

(4) It is necessary to set parameters for scenarios related to actual road conditions.

However, the above information merely provides background information on the present invention and does not correspond to previously disclosed technologies.

In order to solve the problems of the prior art as described above, the present invention accumulates and analyzes events occurring during driving based on driving images obtained through a video information collection device such as a black box pre-mounted in a vehicle, and analyzed An object of the present invention is to provide a method and apparatus for recommending an appropriate driving scenario for a target road based on information.

However, the problem to be solved by the present invention is not limited to the above-mentioned problems, and other problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

A method according to an embodiment of the present invention for solving the above problems is a method performed in an electronic device to recommend a driving scenario of a vehicle. an analysis step of analyzing driving statistics including an object movement trajectory and driving events; A first recommendation step of recommending a scenario section and a driving scenario applied to each scenario section for a target road requiring scenario setting based on the driving statistics; and a second recommendation step of recommending a parameter suitable for the driving scenario for each recommended scenario section based on the driving statistics.

The analyzing step may include detecting an object from the driving image; extracting a movement trajectory for each detected object; and filtering and correcting the extracted movement trajectory for each object, and analyzing the driving statistics reflecting the filtering and correction according to the location included in the driving image.

In the analyzing step, de-identification processing is selectively performed on a de-identification region in the driving image, and when the de-identification processing is performed, the driving statistics are analyzed from the de-identification-processed driving image. It can be.

The non-identification area may include a license plate area of a vehicle, a face area of a pedestrian, or a sign area of a street.

In the step of extracting the movement trajectory, movement trajectories may be extracted only for objects capable of movement among objects in the driving image.

The movement trajectory may be a relative movement trajectory of the movable object based on the vehicle capturing the driving image.

The movable object may include a vehicle, a pedestrian, a bicycle, a motorcycle, or an electric kickboard.

In the extracting of the moving trajectory, analysis of the moving trajectory for the corresponding section may be stopped when the vehicle captured in the driving image deviates from the lane or accelerates or decelerates by more than a first threshold.

In the step of extracting the movement trajectory, when the vehicle captured in the driving image deviates from the lane or accelerates or decelerates by more than a second threshold value greater than the first threshold value, the movement trajectory may be analyzed for the corresponding section. there is.

In the analyzing and storing, the driving event included in the driving statistics may be stored as an event generated by the photographing vehicle.

The driving statistics may include items of weather conditions, road conditions, or unexpectedly appearing objects.

The first recommendation step may include determining the location of the target road; calculating an occurrence distribution for each type of driving event according to a plurality of locations within a location area corresponding to the identified location of the target road among the driving statistics; finding a boundary between the calculated occurrence distributions; Dividing the scenario section according to the found boundary and recommending a driving scenario by using an occurrence distribution for each driving event type included in the section for each divided scenario section.

In the step of finding the boundary, the boundary may be found using driving event types having the highest occurrence distribution.

The calculating step normalizes the driving events within the location area to generate a probability density function for the occurrence distribution for each driving event type, and the finding step generates the generated probability distribution function. can be used to find the boundary.

In the finding step, the boundary may be found using the occurrence distribution to which a weight is applied according to the driving event type.

The second recommendation step may include determining an average number of objects in a corresponding section for each scenario section; and recommending a parameter including the number of identified objects as a parameter for a driving scenario of a corresponding section.

A method according to another embodiment of the present invention is a method performed by first and second electronic devices to recommend a driving scenario of a vehicle, wherein the first electronic device determines a location from a driving image including photographed location information. an analysis step of analyzing driving statistics including the number of objects, object movement trajectories, and driving events according to; transmitting the driving statistics from the first electronic device to the second electronic device; a first recommendation step of recommending, in the second electronic device, a scenario section and a driving scenario applied to each scenario section for a target road requiring scenario setting based on the driving statistics; and a second recommendation step of recommending, in the second electronic device, a parameter suitable for a driving scenario for each recommended scenario section based on the driving statistics.

The method according to one embodiment or another embodiment of the present invention further includes inputting a user's selection or modification of the recommended scenario section and driving scenario between the first and second recommendation steps. In the second recommending step, a parameter suitable for the driving scenario for each scenario section may be recommended based on the driving statistics.

The method according to one embodiment or another embodiment of the present invention may further include, after the second recommendation step, inputting a user's selection or correction of the recommended parameter.

An apparatus according to an embodiment of the present invention is an apparatus for recommending driving scenarios of a vehicle, comprising: a memory storing a driving image including location information of a photographed location; and a control unit performing control using the driving image.

The control unit controls to analyze driving statistics including the number of objects according to locations, object movement trajectories, and driving events from the driving image, and driving scenarios applied for each scenario section and scenario section for a target road requiring scenario setting may be controlled to recommend each based on the driving statistics, and a parameter suitable for a driving scenario for each recommended scenario section may be controlled to be recommended based on the driving statistics.

An apparatus according to another embodiment of the present invention is a device for recommending vehicle driving scenarios, and receives driving statistics including the number of objects according to locations, object movement trajectories, and driving events from a driving image including location information of shooting. communication department; and a control unit configured to recommend driving scenarios and parameters for target roads requiring scenario setting using the received driving statistics.

The control unit controls to recommend a scenario section and a driving scenario applied to each scenario section for the target road based on the driving statistics, respectively, and sets a parameter suitable for the driving scenario for each recommended scenario section to the driving statistics. It can be controlled to recommend based on

The apparatus according to one embodiment or another embodiment of the present invention may further include an input unit through which a user's selection or modification of the recommended scenario section and driving scenario is input.

The control unit may control to recommend a parameter suitable for a driving scenario for each input scenario section based on the driving statistics.

The present invention configured as described above has an advantage of accumulating and analyzing events occurring during driving based on driving images, and recommending an appropriate driving scenario for a target road based on the analyzed information.

That is, the present invention can help derive an appropriate driving scenario in a driving simulation using a computer or a vehicle test environment in an actual road driving environment.

In addition, the present invention has an advantage that it can be used for development and driving tests of autonomous vehicles using image information or various sensor information.

In addition, the present invention has the following differences from the prior art while solving the problems of the prior art as described above.

(1) Statistics on driving events can be provided to analyze potential accident risk factors and road congestion trigger factors that occur in daily driving situations for a given road environment and location.

(2) It is possible to recommend boundary sections for setting scenarios suitable for a given road environment, and it is possible to recommend appropriate scenarios by location and time for scenario design related to actual road conditions.

(3) Scenario parameters related to actual road conditions may be recommended according to road locations and time zones.

In particular, the present invention has the following advantages in terms of practical use compared to the prior art.

(1) Hardware costs for securing data can be reduced.

That is, according to the present invention, it is not necessary to mount expensive additional hardware equipment such as LIDAR or LADAR in a vehicle in order to recommend scenarios and secure driving resources. Instead, the present invention may utilize image data obtained from a vehicle image collection device such as a vehicle black box, which is already installed in many vehicles.

(2) It is easy to secure relevant data.

In other words, the present invention can use a conventional image collection device as it is, so it is relatively easy to secure data. In addition, the present invention does not require the image collection device to be installed in a special location in the vehicle.

The effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 shows a block configuration diagram of a driving scenario recommendation apparatus 100 according to an embodiment of the present invention.

2 shows an example of setting a driving scenario in a road section (target road) requiring scenario setting in relation to the present invention.

3 and 4 show various examples of driving events that can be estimated through driving images.

5 shows a block diagram of the controller 150 in the driving scenario recommendation device 100 according to an embodiment of the present invention.

6 shows a flow chart of a recommendation method according to an embodiment of the present invention.

7 shows a more detailed flow chart for S110.

8 illustrates an example of detecting an object and extracting a movement trajectory from a driving image.

9 is an example showing driving statistics as a table.

10 is an example showing basic information used to extract driving events of FIG. 9 as a table.

11 shows a more detailed flow chart for S120 and S130.

12 illustrates various examples of setting boundaries between occurrence distributions for each type of driving event.

13 is an example of a target road.

FIG. 14 is an example of a result of performing the first and second recommendation functions of the present invention with respect to FIG. 13 .

The above objects and means of the present invention and the effects thereof will become clearer through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art to which the present invention belongs can easily understand the technical idea of the present invention. will be able to carry out. In addition, in describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms in some cases unless otherwise specified in the text. In this specification, terms such as "comprise", "include", "provide" or "have" do not exclude the presence or addition of one or more other elements other than the mentioned elements.

In this specification, terms such as “or” and “at least one” may represent one of the words listed together, or a combination of two or more. For example, "A or B" and "at least one of A and B" may include only one of A or B, or may include both A and B.

In this specification, descriptions following "for example" may not exactly match the information presented, such as cited characteristics, variables, or values, tolerances, measurement errors, limits of measurement accuracy and other commonly known factors. It should not be limited to the embodiments of the present invention according to various embodiments of the present invention with effects such as modifications including.

In this specification, when a component is described as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but there may be other components in the middle. It should be understood that it may be On the other hand, when a component is referred to as 'directly connected' or 'directly connected' to another component, it should be understood that no other component exists in the middle.

In the present specification, when an element is described as being 'on' or 'in contact with' another element, it may be in direct contact with or connected to the other element, but another element may be present in the middle. It should be understood that On the other hand, if an element is described as being 'directly on' or 'directly in contact with' another element, it may be understood that another element in the middle does not exist. Other expressions describing the relationship between components, such as 'between' and 'directly between', can be interpreted similarly.

In this specification, terms such as 'first' and 'second' may be used to describe various elements, but the elements should not be limited by the above terms. In addition, the above terms should not be interpreted as limiting the order of each component, and may be used for the purpose of distinguishing one component from another. For example, a 'first element' may be named a 'second element', and similarly, a 'second element' may also be named a 'first element'.

Unless otherwise defined, all terms used in this specification may be used in a meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

The driving scenario recommendation device 100 according to an embodiment of the present invention is a device for recommending a driving scenario and parameters based on a driving image, and may be an electronic device capable of computing or a computing network.

For example, the electronic device includes a desktop personal computer, a laptop personal computer, a tablet personal computer, a netbook computer, a workstation, and a personal digital assistant (PDA). , a general-purpose computing device such as a smart phone, smart pad, or mobile phone, or a dedicated embedded system, but is not limited thereto.

At this time, the driving image is an image captured by an image collecting device such as a black box installed in the vehicle while the vehicle is driving, and may preferably be image information including location information and time information of the vehicle in motion. This is because the location information can be used later when analyzing driving statistics, recommending driving scenarios, and recommending parameters using driving images. That is, in order to make the most of the functions provided by the present invention, it is preferable that the driving video includes location information. Recent in-vehicle image collection devices are very easy and common to acquire related information in conjunction with location information acquisition devices such as GPS (Global Positioning System) receivers. Of course, the driving image may be captured from one vehicle or from multiple vehicles.

2 shows an example of setting a driving scenario in a road section (target road) requiring scenario setting in relation to the present invention. 3 and 4 show various examples of driving events that can be estimated through driving images.

2 shows that a driving scenario is configured for all target roads (road sections) in which straight and curved sections are combined. In this case, assuming that the entire driving section is divided into four subsections A, B, C, and D, each partial driving scenario may be applied to A, B, C, and D. Of course, the road shape of FIG. 2 is only an example, and the target road may have various road shapes other than FIG. 2 . That is, the number of subsections of the target road may vary, and the scenario application situation may vary for each subsection.

Here, a driving scenario may be defined as a set of possible driving events that may occur during driving. For example, individual events such as "cutting in", "pedestrian crossing", and "lane width reduction" may constitute a driving scenario. In addition, it is also possible to configure an integrated driving scenario by gathering driving scenarios. It may be desirable to establish such driving scenarios by applying relevant standards established by organizations such as ASAM (Association for Standardization of Automation and Measuring Systems).

For example, in section A, “a vehicle in one lane cuts into the second lane in a straight section of lane”, in section B, “a vehicle traveling in two lanes changes to a single lane in a curved section of lane”, in section C, “jaywalking in front Partial driving scenarios such as "appearance of a pedestrian" and "a stopped accident vehicle appears in front when driving at 90 km/h on a straight section on a lane highway" can be applied to section D.

Driving scenarios in the present invention can be provided hierarchically according to the degree of abstraction and concreteness. For example, the highest level describes "traffic congestion", the next level describes "traffic congestion due to heavy rain on the way to work", and the next level describes "traffic congestion due to decrease in public transport users and increase in private car use due to heavy rain on the way to work". At the next level, abstracted scenarios such as "traffic congestion due to a sharp increase in cut-in vehicles in the bottleneck section for entering the highway due to a decrease in public transportation users and an increase in private car use due to heavy rain on the way to work" are abstracted into concrete scenarios. scenarios can be extended. Hereafter, in this specification, embodiments of scenarios are abstractly expressed as "traffic congestion". However, this is in consideration of brevity of expression, and is not limited thereto, and specific scenarios can be provided according to various levels or hierarchies.

On the other hand, since a driving scenario is a very broad concept that is a set of driving events that may occur on a real road, the present invention is limited to a set of driving events that can be collected from the viewpoint of an arbitrary ego vehicle. do. In this case, the vehicle refers to a vehicle in which an image collecting device is installed to capture a driving image.

That is, in the present invention, unlike the conventional method that assumes that the structure of the road network, the location of traffic lights, and the location of crosswalks are known from the point of view of an omniscient observer, driving that can be estimated using only image information that can be collected through the Ego vehicle Only incidents are considered. Despite these limitations, according to the present invention, since the location and time information of each driving event is simultaneously recorded, it is later utilized in connection with information such as traffic accident statistics by location or detailed maps. Of course, the range can be widened.

3 and 4 show some examples of driving events that can be collected from any Ego vehicle perspective described above. That is, from the perspective of the driver in the vehicle or the camera installed in the vehicle, "stopping", "appearance of accident vehicle ahead", "stopping on the shoulder and appearance of accident vehicle", "reverse driving", "cutting in", "lane reduction and Cut In", "Curve Cut In", "Pedestrian Crossing", "Bicycle Crossing", "Motorcycle Cut In", "Multiple Pedestrian Crossing", "Interrupting At Merge", "Leave Separation" , driving events such as "overtaking", "overtaking after crossing the center line", etc. may be acquired.

In the present invention, it is possible to continuously and gradually find out which driving events frequently occur for a specific location and time interval and what statistical characteristics the distribution of objects such as vehicles or pedestrians have according to location or time, It has a feature that can be processed and provided over time. To this end, driving image information including location and time information is received and used. Of course, such driving image information may be collected from various vehicles.

Meanwhile, the recommendation device 100 may largely provide three functions, and these three functions may be provided independently or dependently.

First, the recommendation device 100 may provide an analysis function that analyzes driving statistics including the number of objects according to location and time, object movement trajectories, and driving events from a driving image including location and time information of a photograph. Of course, this analysis function may be provided by a separate electronic device (hereinafter referred to as “analysis device”) other than the recommendation device 100, and in this case, driving statistics analyzed by the analysis device are transferred to the recommendation device 100. can be conveyed For example, driving statistics analyzed by the analysis device may be stored in a separate database system, and the recommendation device 100 may access the corresponding database system and receive corresponding driving statistics. Alternatively, the analysis device may store the analyzed driving statistics, and the corresponding driving statistics may be transmitted to the recommendation device 100 through communication between the analysis device and the recommendation device 100 . However, a detailed description of such an analysis function will be described later in the recommended method of the present invention.

Next, the recommendation device 100 recommends a scenario section and a driving scenario applied to each scenario section for a target road requiring scenario setting based on driving statistics (hereinafter, referred to as a "first recommendation function"). can provide. However, a detailed description of the first recommendation function will be described later in the recommendation method of the present invention.

Next, the recommendation device 100 may provide a function of recommending a parameter suitable for a driving scenario for each recommended scenario section based on the driving statistics (hereinafter, referred to as a "second recommendation function"). However, a detailed description of the second recommendation function will be described later in the recommendation method of the present invention.

For example, the recommendation device 100 can operate as an end-device installed in a vehicle or another vehicle in which an image collection device for capturing driving images is installed, and is not installed in the vehicle, and provides an analysis function and a first recommendation It may be implemented as a system that provides a function or a second recommendation function and may operate as a server that transmits a result of the corresponding analysis or recommendation to a vehicle terminal or the like.

Specifically, as shown in FIG. 1 , the recommendation device 100 may include an input unit 110, a communication unit 120, a display 130, a memory 140, and a controller 150. Of course, the recommendation device 100 may also include an image collection device that captures driving images.

The input unit 110 generates input data in response to various user inputs and may include various input means. For example, the input unit 110 includes a keyboard, a key pad, a dome switch, a touch panel, a touch key, a touch pad, and a mouse. (mouse), menu button (menu button), etc. may be included, but is not limited thereto.

The communication unit 120 is a component that communicates with other devices. For example, the communication unit 120 may perform 5th generation communication (5G), long term evolution-advanced (LTE-A), long term evolution (LTE), Bluetooth, Bluetooth low energy (BLE), near field communication (NFC), Wireless communication such as WiFi communication or wired communication such as cable communication may be performed, but is not limited thereto. For example, the communication unit 120 may receive driving images or driving statistics from other devices, and may transmit analysis results or recommendation results to other devices.

The display 130 is a component that displays various image data on a screen. For example, the display 130 may be composed of a non-emissive panel or a light emitting panel. For example, the display 130 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, and a micro electromechanical system (MEMS). mechanical systems) display, or electronic paper (electronic paper) display, etc. may be included, but is not limited thereto. For example, the display 130 may display driving images, driving statistics, analysis results, or recommendation results on the screen. Also, the display 130 may be combined with the input unit 110 and implemented as a touch screen or the like.

The memory 140 stores various types of information necessary for the operation of the recommendation device 100 . For example, the stored information may include driving images, driving statistics, analysis results, recommendation results, or program information related to a recommendation method described later, but is not limited thereto. For example, the memory 140 may be a hard disk type, a magnetic media type, a compact disc read only memory (CD-ROM), and an optical media type according to its type. ), Magneto-optical media type, Multimedia card micro type, flash memory type, read only memory type, or RAM type (random access memory type), etc., but is not limited thereto. In addition, the memory 140 may be a cache, a buffer, a main memory, an auxiliary memory, or a separately provided storage system depending on its purpose/location, but is not limited thereto.

The controller 150 may perform various control operations of the recommendation device 100 . That is, the controller 150 may control the execution of a recommendation method to be described later, and the rest of the components of the recommendation device 100, that is, the input unit 110, the communication unit 120, the display 130, the memory 140, etc. You can control the action. For example, the control unit 150 may include, but is not limited to, a processor that is hardware or a process that is software that is executed in the corresponding processor. For example, a processor includes a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA). ), etc., but is not limited thereto.

The control unit 150 controls the performance of the recommendation method according to an embodiment of the present invention, and may include an image processing unit 151 and a scenario processing unit 152 as shown in FIG. 5 . For example, the image processing unit 151 and the scenario processing unit 152 may be hardware configurations of the control unit 150 or software processes executed by the control unit 150, but are not limited thereto.

Hereinafter, the recommendation method according to the present invention will be described in more detail.

6 shows a flow chart of a recommendation method according to an embodiment of the present invention, and FIG. 7 shows a more detailed flow chart for S110.

The recommendation method according to the present invention is performed by the driving scenario recommendation device 100 and can be performed under the control of the controller 150, and is a method of selectively performing an analysis function, a first recommendation function, and a second recommendation function. That is, the recommendation method according to an embodiment of the present invention may include S110 to S130 as shown in FIG. 6 .

S110 is a step of providing an analysis function according to the control of the image processing unit 151 of the control unit 150. That is, the image processing unit 151 may control analysis of driving statistics including the number of objects according to location and time, object movement trajectories, and driving events from driving images including location information captured by the image collection device. As shown in FIG. 4 , S110 may include S111 to S116.

In S111, the driving image captured by the image collecting device is input (delivered) to the image processing unit 151. At this time, it is preferable that the driving video includes the captured location information together.

In S112, the image processing unit 151 selectively performs de-identification processing on the input driving image. At this time, the de-identification processing is image processing to de-identify a specific area of the driving video (ie, remove information related to personal information), and this specific area is referred to as a non-identifying area.

In this case, the non-identification area may include a license plate area of a vehicle, a face area of a pedestrian, or a sign area of a street. That is, the image processing unit 151 detects the license plate area of the vehicle in the driving image and performs de-identification processing, detects the pedestrian's face area in the driving image and performs de-identification processing, or detects the signboard area on the street. Thus, de-identification can be performed. For example, the de-identification processing may include, but is not limited to, mosaic processing, blurring processing, and monochromatic processing for the non-identification area.

However, S112 may be selectively performed. That is, the present invention does not store or output personal information even if de-identification processing according to S112 is not performed. Nevertheless, in order to fundamentally exclude issues related to personal information processing in terms of secondary processing of data, S112 may be included.

In S113, the image processing unit 151 detects and recognizes an object in the input driving image. This is a process of detecting the existence and location of various objects that may appear during driving, such as road lanes, cars, pedestrians, bicycles, and traffic lights, in the input driving image. For this detection, various conventional image processing techniques of object detection may be used.

For example, for lane detection, an image processing method such as Hough transform, which extracts a straight line or a curved component from a driving image, may be used. In addition, an artificial neural network-based object detection technique such as Yolo may be used to detect a movable object such as a vehicle, pedestrian, bicycle, motorcycle, or electric kickboard. However, in the present invention, it is not limited to the above-described lane detection method or object detection technique, and various object detection techniques may be used.

8 illustrates an example of detecting an object and extracting a movement trajectory from a driving image. However, in FIG. 8 , the Ego Vehicle, which is the subject of capturing the driving video, does not generally appear in the driving video, but is indicated as a vehicle framed by a dotted line for understanding the concept. On the other hand, in FIG. 8 , roads and vehicles in large rectangles represent acquired driving images.

In S114, the image processing unit 151 extracts a moving trajectory for each detected object. In this case, the movement trajectory means a relative movement trajectory of the object based on the ego vehicle, that is, a relative movement trajectory of the object viewed from the viewpoint of the ego vehicle.

For example, referring to FIG. 8 , except for “vehicle F” painted in black, all vehicles including the vehicle are traveling at a speed of “V _0” . Accordingly, the relative speeds of "Vehicle A", "Vehicle B", "Vehicle C", "Vehicle D", and "Vehicle E" moving at the same speed as vehicle are zero. That is, when a driving video captured by a vehicle is played back, vehicles with a relative speed of 0 do not move in the video, but a random movement trajectory occurs in the case of a black “vehicle F” trying to “interrupt”. Of course, the relative velocities of "Vehicle A", "Vehicle B", "Vehicle C", "Vehicle D" and "Vehicle E" with respect to the vehicles are usually not zero, but for vehicles traveling above a threshold value In the way of selection, it is possible to find a vehicle object with a special movement trajectory. A driving event such as "cutting in" or "overtaking" shown in FIG. 8 can be estimated by synthesizing lane information, vehicle position, and movement trajectory.

On the other hand, in S114, various exceptional situations may occur. For example, an object such as a "traffic light" normally fixed to the ground may appear to move relatively quickly in a driving image acquired due to the movement of an Ego Vehicle. . In order to eliminate this exceptional situation, in S114, movement trajectories can be selectively tracked (extracted) only for objects that are normally movable and important in driving scenario analysis, such as vehicles, pedestrians, bicycles, motorcycles, or electric kickboards. .

In S115, the image processing unit 151 performs filtering and correcting the moving trajectory for each object in order to facilitate scenario analysis in the extracted moving trajectory. For example, it is desirable to remove the movement trajectory of an object detected in an area other than the road (ie, the sky or the surrounding area). In addition, in FIG. 8 , the ego vehicle itself may cause changes such as acceleration or deceleration or lane change. In this case, changes in movement trajectory may occur globally in all vehicles other than the ego vehicle. The purpose of "object movement trajectory correction" is to correct for this case and select the movement trajectory for each object. For correction corresponding to this situation, the following two methods can be selectively used.

(1) If the ego vehicle deviates from the lane or accelerates or decelerates by more than a first predetermined threshold value, the image processing unit 151 stops object extraction or movement trajectory information analysis in the corresponding section. This is to reduce computational processing cost and use only effects other than Ego vehicle for analysis. Of course, the first threshold value applied to lane departure and the first threshold value applied to acceleration and deceleration may be different numerical values.

At this time, lane departure detection of the ego vehicle can be detected using driving image information, and acceleration and deceleration information of the ego vehicle can be identified using GPS information, acceleration sensor information, or gyro sensor information linked to the image collecting device. . If such non-image information is not available, relative acceleration and deceleration may be estimated using the acquired driving image information.

For example, if an object that consistently changes in proportion to the movement of a vehicle, such as the floor of a road, is set as a reference point, the acceleration and deceleration of the ego vehicle can be controlled by continuously calculating the motion vector of the corresponding area over time. can be calculated

(2) If the ego vehicle deviates from the lane more than the second threshold value (greater than the first threshold value) or accelerates and decelerates, it is recorded as a driving event caused by the ego vehicle. For example, if an Ego vehicle causes an event such as "changing lanes from 2 lanes to 3 lanes on a total 3-lane road" through driving video analysis, this is a method of recording. Of course, the second threshold value applied to lane departure and the second threshold value applied to acceleration and deceleration may be different numerical values.

FIG. 9 is an example showing driving statistics in a table, and FIG. 10 is an example showing basic information used to extract driving events of FIG. 9 in a table.

In S116, the image processing unit 151 analyzes and extracts driving statistics according to location and time. In this case, the driving statistics are information including the number of objects detected in the corresponding time zone according to the location and time included in the driving image provided as input to the present invention, the movement trajectory of the objects, and the analyzed driving event. For example, the driving statistics extracted according to FIG. 8 are the number of objects for “lane road” and “total 6 vehicles”, the moving trajectory for these objects, and the driving event analyzed as “cutting in” or “overtaking” can include The driving statistics extracted in this way can be stored and managed in the memory 140, as shown in FIGS. 9 and 10, and can be continuously stored and managed through a separate database or file system. .

In addition, basic information used to extract driving events of driving statistics may also be stored and managed, as shown in FIG. 10 . Information stored and managed in this way may be output through the display 130 or transmitted to other devices through the communication unit 120 .

However, in FIG. 9 , only the number of objects such as <vehicle number> and <pedestrian number> is displayed, but the present invention is not limited thereto. That is, information that can be analyzed through driving images is additionally included in driving statistics, such as weather conditions such as rain or snow, road surface conditions such as damaged areas or icy areas, and the sudden appearance of objects such as gravel. Of course, it can be stored and managed.

Meanwhile, S110 may be performed under the control of a control unit of another electronic device, and information on driving statistics, which is an analysis result, may be transmitted to the tracking device 100 through the communication unit 120.

11 shows a more detailed flow chart for S120 and S130.

Next, S120 is a step of providing a first recommendation function under the control of the scenario processing unit 152 of the controller 150, and may include S121 to S125 as shown in FIG. 11 . Further, S130 is a step of providing a second recommendation function under the control of the scenario processing unit 152 of the controller 150. As shown in FIG. 11, S120 may include S121 to S125, and S130 may include S131 to S131 to S125. S134 may be included.

In S121, the scenario processing unit 152 receives information about a target road for which scenario setting is required from the user. Accordingly, the scenario processing unit 152 may determine the location of the target road. Of course, it is preferable that the information on the target road is used by applying the conventional road information display format that is widely used to include the network and location of the road. For example, the information on the target road may include identifier information and location information of the corresponding target road.

In S122, the scenario processing unit 152 divides the entire section of the target road into subsections (ie, scenario sections) based on the location and hourly driving statistical information obtained through S110 and recommends a driving scenario applied to each scenario section. do.

The specific method used in S122 is as follows.

(1) First, the scenario processing unit 152 identifies a location area corresponding to the location information of a target road in previously stored and managed driving statistics. For example, a corresponding location area may be identified using a query including location information of a target road. Of course, in some cases, the search may be performed by using time information for scenario application together with location information of the target road.

(2) The scenario processing unit 152 calculates an occurrence distribution for each type of driving event according to a plurality of locations within a location area corresponding to the location of the target road identified among driving statistics. That is, the occurrence distribution (or frequency) of driving events included in the location area of the target road among driving statistics is calculated according to the location.

(3) The scenario processing unit 152 uses the calculated occurrence distribution for each type of driving event to find a boundary between occurrence distributions. For example, the scenario processing unit 152 may find a boundary between occurrence distributions for each driving event type so as to preferentially recommend a driving event having a high frequency. There may be various ways to find such a boundary, but in the present invention, one of the following three methods can be selectively used according to the application.

(3-1) The scenario processing unit 152 calculates an occurrence distribution for each driving event type calculated for the target road, and determines a boundary between the distributions as a section to which the scenario is applied. 12 corresponds to an example of this, and it is assumed that the occurrence distribution of four driving events is calculated for a given target road.

In FIG. 12A , in the case of “Incident 1”, the highest occurrence distribution is shown in the first part of the target road, and “Incident 2” shows the highest occurrence distribution between the beginning part and the middle part of the target road. Using such occurrence distribution information, it is possible to group and extract events showing the highest occurrence distribution at each location of a given target road.

In FIG. 12A, "event 1" is between the driving sections (P1 and P2), "event 2" is between the driving sections (P2 and P3), and "event 3" is between the driving sections (P3 and P4). Between (P4, P5), "event 4" shows the highest occurrence distribution, respectively. Therefore, the scenario processing unit 152 may determine the boundaries between occurrence distributions as {P1, P2, P3, P4, P5}, and set the intervals between them as scenario sections, respectively. A scenario related to a driving event with a high occurrence frequency (probability) may be determined as a driving scenario for a corresponding scenario section.

(3-2) Similar to (3-1) described above, but the scenario processing unit 152 normalizes the occurrence distribution for each driving event type with respect to the target road, and calculates the probability distribution function for the occurrence distribution for each driving event type. (probability density function), and then find the boundary between the driving event distributions in the corresponding probability distribution function. This is shown in Figure 12b. In this case, the driving location information of {P1, P2, P3, P4, P5} may have a different value from the location information of FIG. 12A. Accordingly, the scenario processing unit 152 may determine {P1, P2, P3, P4, P5}, which is different from the case of FIG. 12A, as the boundary between occurrence distributions, and set it as a scenario section. At this time, each scenario section in driving statistics A scenario related to a driving event with the highest occurrence frequency (probability) may be determined as a driving scenario for a corresponding scenario section.

(3-3) Similar to (3-1) described above, but the scenario processing unit 152 transforms the occurrence distribution by differentiating weights for each driving event type for the target road, and then converts the occurrence distribution between the event distributions. find the boundary This is shown in Figure 12c. In this case, the driving location information of {P1, P2, P3, P4, P5} may have a different value from the location information of FIG. 12a. Accordingly, the scenario processing unit 152 may determine {P1, P2, P3, P4, P5}, which is different from the case of FIG. 12A, as the boundary between occurrence distributions, and set it as a scenario section. At this time, each scenario section in driving statistics A scenario related to a driving event with the highest occurrence frequency (probability) may be determined as a driving scenario for a corresponding scenario section.

The advantage of using the method (3-3) is that, in addition to the driving statistical information collected according to S110, external information or a weight value thought by the user can be applied. For example, if "Incident 1" and "Incident 4" have a greater potential risk, or if more occurrences are expected in the future, a higher weight may be given to these driving events to induce recommendations for driving scenarios related to them. there is.

In the case of these weighting functions, the urgency of the event, the risk of the event (criticality), the time budget of the event, the driver's mental workload for the event, and the driver's physical burden for the event (Physical workload), etc. can be reflected, so it can be said to be a very meaningful function.

(3-4) In the case of (3-1) to (3-3) described above, it is a method of determining the value with the highest probability as the driving scenario for the corresponding scenario section by looking at the occurrence distribution. However, even if the frequency of occurrence is low, there may be meaningful driving events. Therefore, this method is a method for providing related information upon request even for driving events having a lower frequency occurrence distribution. For example, in the scenario section between (P1 and P2) in FIG. 12A, “event 1” has the highest occurrence distribution and “event 2” has the next highest occurrence distribution. In this case, with respect to the scenario section between (P1 and P2), driving statistical information for "Incident 2" may be provided in addition to "Incident 1".

After that, the user may use the recommended part as it is or modify it. That is, in S123, whether to select a driving scenario for each recommended scenario section is input through the input unit 110. When the corresponding recommendation is selected, the driving scenario for each recommended scenario section is determined as the driving scenario (S125). On the other hand, when the corresponding recommendation is not selected, correction information for the recommended scenario section and driving scenario is input through the input unit 110 in S124. In this case, the driving scenario for each scenario section modified by the user is determined as the driving scenario (S125).

In S131, since the driving scenario for each scenario section is determined, the scenario processing unit 152 recommends a parameter suitable for the driving scenario for each scenario section based on driving statistics. For example, specific figures such as the number of lanes, the number of driving vehicles, and the number of pedestrians can be recommended for each scenario section.

The specific method used in S131 is as follows.

(1) First, when a scenario section and a recommended scenario are determined according to S125, the scenario processing unit 152 uses location information as a query for the corresponding scenario section to obtain objects from pre-stored information as shown in FIG. 9 of driving statistics. Search.

(2) After that, the scenario processing unit 152 determines the average number of objects in a corresponding section for each scenario section. That is, the occurrence distribution (or frequency) of objects can be calculated for each scenario section.

(3) The scenario processing unit 152 recommends parameters including the number of identified objects. That is, it is possible to recommend parameters suitable for the recommended scenario according to the object occurrence distribution for each scenario section of the target road.

For example, since a driving scenario related to “Incident 1” may be recommended in the scenario section between (P1 and P2) in FIG. 12A, the number of objects (eg, “number of vehicles”, “number of pedestrians”, etc.) ) can be obtained through a search by location. If this is averaged over the corresponding scenario section or target road, a parameter including the average number of objects (eg, "average number of vehicles", "average number of pedestrians", etc.) may be provided as a parameter for the corresponding scenario section. .

After that, the user can use the recommended parameters as they are or modify them. That is, in S132, whether to select parameters for each recommended scenario section is input through the input unit 110. When the corresponding recommendation is selected, parameters for each recommended scenario section are determined as parameters (S134). On the other hand, when the corresponding recommendation is not selected, correction information for parameters for each recommended scenario section is input through the input unit 110 in S133. In this case, parameters for each scenario section modified by the user are determined as parameters (S134).

If the recommendation method according to the present invention is used, great help can be provided when configuring a scenario in a virtual simulator on a computer or a test bed environment in which a real car can be driven and tested.

13 is an example of a target road, and FIG. 14 is an example of a result of performing the first and second recommendation functions of the present invention with respect to FIG. 13 .

For example, it is assumed that a subject who intends to construct a scenario provides information about a target road as shown in FIG. 13 . In this case, only network or location information on the target road is provided, and it is unknown which driving scenario to apply. In particular, let's assume that there is no objective information about which scenario parameters to set.

In this case, the term “driving scenario” may be a combination of various driving events such as “vehicle cutting in”, “overtaking”, and “pedestrian appearing” that may occur on the road, as previously defined. In addition, in setting up a scenario including a driving event such as "vehicle cutting in", "parameter" means "how many vehicles around", "whether there were pedestrians around", "commuting time Awareness", "Is it snowing?" It means a number related to detailed settings.

According to the present invention, a driving image for a location related to a target road for which a driving scenario recommendation is required is analyzed in advance and then used, or when there is no corresponding driving image, an image collecting device such as a black box is directly used for the corresponding driving section. Thus, the captured driving video can be used. In this case, it is desirable to repeatedly drive the same route to sufficiently secure data. The present invention analyzes driving image information including collected location information, estimates objects appearing on the road, movement trajectories of objects, and driving events occurring during driving, and stores and manages them as driving statistics.

As a result, the present invention recommends driving scenarios and scenario parameters for each scenario section suitable for the target road. FIG. 14, as an example of the recommended information, first provides location information {P1, P2, P3, P4, P5} for determining a scenario section. also. In each scenario section, an appropriate driving event is recommended. In the case of driving section A, the section is determined by (P1, P2), “interrupting” as the first priority, “overtaking” as the second priority, and “average number of vehicles” and “average number of vehicles” as scenario parameters. Pedestrian count" was provided. Based on such recommended information, a computer simulation environment may be configured or an actual driving test may be conducted.

The present invention configured as described above has an advantage of accumulating and analyzing events occurring during driving based on driving images, and recommending an appropriate driving scenario for a target road based on the analyzed information. That is, the present invention can help derive an appropriate driving scenario in a driving simulation using a computer or a vehicle test environment in an actual road driving environment.

In particular, the present invention has the following differences from the prior art while solving the problems of the prior art as described above.

(1) Based on the analysis function, statistics on driving events can be provided to analyze potential accident risk factors and road congestion trigger factors that occur in daily driving situations for a given road environment and location. .

(2) Based on the first recommendation function, it is possible to recommend boundary sections for setting scenarios suitable for a given road environment, and to design scenarios related to actual driving road conditions, appropriate scenarios by location and time can be recommended.

(3) Based on the second recommendation function, scenario parameters associated with actual road conditions may be recommended according to road locations and time zones.

(1) Hardware costs for securing data can be reduced.

(2) It is easy to secure related data.

For example, in the case of a conventional 3D precision map collection device, data collection costs and entry barriers for securing data existed because specially manufactured image collection devices were acquired by mounting them on a small number of dedicated collection vehicles. On the other hand, if the present invention is used, anyone can take a driving video in a general vehicle. Therefore, it is possible to obtain driving image information by directly shooting or purchasing an image, and through this, driving scenarios and parameters can be recommended.

For reference, the reason why the driving scenario analysis in the present invention does not have much difficulty even with only the image quality through the vehicle black box is that accurate and precise values related to the vehicle position or speed are required for the scenario analysis required by the present invention. because it does not Instead, situations such as relative movement between vehicles or sudden changes are used as important. For example, information such as "how the distance to the car in front or behind changes relative", "whether the vehicle cuts in while driving", "the number of lanes", "weather conditions", and "how dense the vehicles are" will be mainly used

In the detailed description of the present invention, specific embodiments have been described, but various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, and should be defined by the following claims and equivalents thereof.

The present invention relates to a method and apparatus for recommending a driving scenario, which accumulates and analyzes events occurring during driving based on driving images, and provides the analyzed information to a method and apparatus for recommending an appropriate driving scenario for a target road. Therefore, there is industrial applicability.

Claims

A method performed in an electronic device to recommend a driving scenario of a vehicle, comprising:

an analysis step of analyzing driving statistics including the number of objects according to locations, object movement trajectories, and driving events from the driving image including location information of the photographed location;

A first recommendation step of recommending a scenario section and a driving scenario applied to each scenario section for a target road requiring scenario setting based on the driving statistics; and

a second recommendation step of recommending a parameter suitable for a driving scenario for each recommended scenario section based on the driving statistics;

How to include.
According to claim 1,

The analysis step is

detecting an object in the driving image;

extracting a movement trajectory for each detected object; and

filtering and correcting the extracted movement trajectory for each object, and analyzing the driving statistics reflecting the filtering and correction according to a location included in the driving image;

How to include.
According to claim 2,

In the analyzing step, de-identification processing is selectively performed on a de-identification region in the driving image, and when the de-identification processing is performed, the driving statistics are analyzed from the de-identification-processed driving image. how to be
According to claim 3,

The non-identification area includes a license plate area of a vehicle, a face area of a pedestrian, or a sign area of a street.
According to claim 2,

The step of extracting the movement trajectory is a method of extracting a movement trajectory only for objects capable of movement among objects in the driving image.
According to claim 5,

The movement trajectory is a relative movement trajectory of the movable object based on the vehicle capturing the driving image.
According to claim 5,

The movable object includes a vehicle, a pedestrian, a bicycle, a motorcycle, or an electric kickboard.
According to claim 2,

The extracting of the moving trajectory stops the analysis of the moving trajectory for the corresponding section when the vehicle captured in the driving image deviates from the lane or accelerates or decelerates by more than a first threshold value.
According to claim 8,

The step of extracting the movement trajectory analyzes the movement trajectory for the corresponding section when the vehicle captured in the driving image departs from the lane or accelerates or decelerates by more than a second threshold value greater than the first threshold value,

In the analyzing and storing, a driving event included in the driving statistics is stored as an event generated by the photographing vehicle.
According to claim 1,

The driving statistics include a weather condition or a road surface condition or an item of an unexpectedly appearing object.
According to claim 1,

In the first recommendation step,

locating the target road;

calculating an occurrence distribution for each type of driving event according to a plurality of locations within a location area corresponding to the identified location of the target road among the driving statistics;

finding a boundary between the calculated occurrence distributions; and

dividing the scenario section according to the found boundary and recommending a driving scenario by using an occurrence distribution for each driving event type included in the section for each divided scenario section;

How to include.
According to claim 11,

The step of finding the boundary is a method of finding the boundary using driving event types having the highest occurrence distribution.
According to claim 11,

The calculating step normalizes the driving events within the location area to generate a probability density function for an occurrence distribution for each driving event type;

The finding step is a method of finding the boundary using the generated probability distribution function.
According to claim 11,

The finding step is a method of finding the boundary using the occurrence distribution to which a weight is applied according to the driving event type.
According to claim 1,

In the second recommendation step,

determining an average number of objects in a corresponding section for each scenario section; and

recommending a parameter including the number of identified objects as a parameter for a driving scenario of a corresponding section;

How to include.
A method performed in first and second electronic devices to recommend a driving scenario of a vehicle, comprising:

an analysis step of analyzing, by the first electronic device, driving statistics including the number of objects according to locations, object movement trajectories, and driving events from the driving image including location information of the photographed location;

transmitting the driving statistics from the first electronic device to the second electronic device;

a first recommendation step of recommending, in the second electronic device, a scenario section and a driving scenario applied to each scenario section for a target road requiring scenario setting based on the driving statistics; and

a second recommendation step of recommending, by the second electronic device, a parameter suitable for a driving scenario for each recommended scenario section based on the driving statistics;

How to include.
According to claim 16,

Between the first and second recommendation steps, further comprising inputting a user's selection or modification of the recommended scenario section and driving scenario;

The second recommendation step is a method of recommending a parameter suitable for a driving scenario for each of the input scenario sections based on the driving statistics.
A device for recommending a driving scenario of a vehicle,

a memory storing a driving image including location information of the shooting; and

Including; a control unit for performing control using the driving image,

The control unit,

Control to analyze driving statistics including the number of objects according to location, object movement trajectories, and driving events from the driving image;

Control to recommend a scenario section and a driving scenario applied to each scenario section for a target road requiring scenario setting based on the driving statistics,

An apparatus for controlling to recommend a parameter suitable for a driving scenario for each recommended scenario section based on the driving statistics.
A device for recommending a driving scenario of a vehicle,

a communication unit configured to receive driving statistics including the number of objects according to location, object movement trajectories, and driving events from the driving image including the location information of the photographed location; and

A control unit configured to recommend driving scenarios and parameters for target roads requiring scenario setting using the received driving statistics;

The control unit,

For the target road, control to recommend scenario sections and driving scenarios applied to each scenario section based on the driving statistics, respectively;

An apparatus for controlling to recommend a parameter suitable for a driving scenario for each recommended scenario section based on the driving statistics.
According to claim 19,

Further comprising an input unit for inputting a user's selection or modification of the recommended scenario section and driving scenario;

The control unit controls to recommend a parameter suitable for a driving scenario for each of the input scenario sections based on the driving statistics.