KR20210100775A - Autonomous driving device for detecting road condition and operation method thereof - Google Patents

Abstract

Various embodiments of the present invention relate to an autonomous driving device for detecting a road condition and an operation method thereof. The autonomous driving device of the present invention comprises: first and second sensors included in a lamp of an autonomous driving vehicle; and a processor electrically connected to the first and second sensors, wherein the processor controls the autonomous driving vehicle such that the autonomous driving vehicle checks a free zone at the front side, based on first sensing data obtained by the first sensor, obtains second sensing data for the free zone, based on the second sensor, generates map information for indicating a height of a road, based on the second sensing data, and predicts a road condition, based on a pattern of the map information. According to the present invention, one or more of the autonomous driving vehicle, a user terminal, and a server can be linked to an artificial intelligence module, an unmanned aerial vehicle (UAV) such as a drone, a robot, an augmented reality (AR) device, a virtual reality (VR) device, a 5G service-related device, and the like.

Description

Autonomous driving device for detecting road surface condition and operating method thereof

Various embodiments of the present disclosure relate to an autonomous driving apparatus for controlling autonomous driving of a vehicle and an operating method thereof, and more particularly, to predicting a road surface condition using sensing data obtained through a camera and a lidar in a vehicle. It's about technology.

The autonomous driving vehicle refers to a vehicle having a function that can drive itself without a user's manipulation. The autonomous vehicle transmits information obtained based on a camera, sensor and/or headlamp attached to the vehicle to the server, and receives information about the conditions of surrounding vehicles and roads from the server, thereby automatically driving without user intervention. can be performed.

Such an autonomous driving vehicle may detect a lane, a vehicle, a pedestrian, a road surface, etc. as a main detection target for autonomous driving.

In general, an autonomous vehicle may detect a road surface by obtaining a virtual road surface plane function and fitting it to a tertiary plane that divides road surface points. However, such a road surface detection method has a problem in that it cannot detect a road surface condition such as a wet state, a road surface state immediately after snowfall, a speed bump installed state, a pothole state, and the like.

SUMMARY OF THE INVENTION An object of the present disclosure is to provide an autonomous driving apparatus for predicting a state of a road surface using sensing data acquired through a camera and a lidar in a vehicle, and an operating method thereof.

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those of ordinary skill in the art to which the present disclosure belongs from the description below. There will be.

An autonomous driving device according to various embodiments of the present disclosure includes a first sensor and a second sensor included in a lamp of an autonomous driving vehicle, and a processor electrically connected to the first sensor and the second sensor, the processor comprising: The autonomous vehicle checks a front free area based on first sensing data obtained through the first sensor, obtains second sensing data for the free area based on the second sensor, and the second sensing It is possible to generate map information expressing the height of the road surface based on the data, and control to predict the road surface condition based on the pattern of the map information.

A method of operating an autonomous driving apparatus according to various embodiments of the present disclosure includes an operation of identifying a free area in front based on first sensing data through a first sensor included in a lamp of an autonomous driving vehicle, and a second sensor included in the lamp. obtaining second sensing data for the free area based on and predicting a road surface condition based on the pattern of the map information.

The autonomous driving apparatus according to the embodiments of the present disclosure may improve the prediction performance of the road surface condition by checking the free area from the image obtained through the camera and generating map information expressing the height of the road surface with respect to the free area using the lidar. .

Effects obtainable in the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art to which the present disclosure belongs from the description below. will be.

1 shows an example of basic operations of an autonomous vehicle and a 5G network in a 5G communication system.
1 shows an example of basic operations of an autonomous vehicle and a 5G network in a 5G communication system.
2 shows an example of an application operation of an autonomous vehicle and a 5G network in a 5G communication system.
3 to 6 show an example of an autonomous driving vehicle operation using 5G communication.
7A illustrates sensors included in a headlamp of a vehicle according to various embodiments of the present disclosure;
7B illustrates a field of view (FOV) of sensors in a headlamp according to various embodiments.
8 is a control block diagram of a vehicle according to various embodiments of the present disclosure;
9 is a diagram conceptually illustrating an electronic device according to various embodiments of the present disclosure;
10A to 10C are diagrams for explaining an operation of generating a grid map for a road surface in an electronic device according to various embodiments of the present disclosure;
11 is a flowchart illustrating a method for predicting a road surface condition in an electronic device according to embodiments of the present disclosure;
12 is a flowchart illustrating a method of generating map information representing a height of a road surface in an electronic device according to embodiments of the present disclosure;
13 is a flowchart illustrating a method of generating a grid for a free region in an electronic device according to embodiments of the present disclosure;
14 is a flowchart illustrating a method of controlling a vehicle based on a road surface condition in an electronic device according to embodiments of the present disclosure;
15 is a flowchart illustrating a method of analyzing an abnormal section of a road surface in an electronic device according to embodiments of the present disclosure;

Advantages and features of the present disclosure, and a method of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present disclosure is not limited to the following embodiments, but will be implemented in various different forms, and only the present embodiments allow the disclosure of the present disclosure to be complete, and to those of ordinary skill in the art to which the present disclosure belongs It is provided to fully understand the scope of the disclosure, which is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

When one component is referred to as “connected to” or “coupled to” with another component, it means that it is directly connected or coupled to another component or intervening another component. including all cases. On the other hand, when one component is referred to as “directly connected to” or “directly coupled to” with another component, it indicates that another component is not interposed therebetween. “and/or” includes each and every combination of one or more of the recited items.

Terms used in the present disclosure are for describing the embodiments and are not intended to limit the present disclosure. In this disclosure, the singular also includes the plural unless the phrase specifically dictates otherwise. As used herein, “comprises” and/or “comprising” refers to a referenced component, step, operation and/or element of one or more other components, steps, operations and/or elements. The presence or addition is not excluded.

Although the first, second, etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another.

Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present disclosure. Unless otherwise defined, all terms (including technical and scientific terms) used in the present disclosure may be used with meanings commonly understood by those of ordinary skill in the art to which this disclosure belongs. Also, commonly used terms defined in the dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

As used herein, the term 'unit' or 'module' means software or a hardware component such as an FPGA or ASIC, and the 'unit' or 'module' performs certain roles. However, 'part' or 'module' is not meant to be limited to software or hardware. A 'unit' or 'module' may be configured to reside on an addressable storage medium or may be configured to reproduce one or more processors. Thus, as an example, 'part' or 'module' refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, may include procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Components and functionality provided in 'units' or 'modules' may be combined into a smaller number of components and 'units' or 'modules' or additional components and 'units' or 'modules' can be further separated.

Steps of a method or algorithm described in connection with some embodiments of the present disclosure may be directly implemented in hardware executed by a processor, a software module, or a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of recording medium known in the art. An exemplary recording medium is coupled to the processor, the processor capable of reading information from, and writing information to, the storage medium. Alternatively, the recording medium may be integral with the processor. The processor and recording medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal.

1 shows an example of basic operations of an autonomous vehicle and a 5G network in a 5G communication system.

The autonomous vehicle transmits specific information transmission to the 5G network (S1). The specific information may include autonomous driving-related information. The autonomous driving-related information may be information directly related to driving control of a vehicle. For example, the autonomous driving-related information may include one or more of object data indicating objects around the vehicle, map data, vehicle state data, vehicle location data, and driving plan data. .

The autonomous driving-related information may further include service information required for autonomous driving. For example, the specific information may include information about the destination and the vehicle's stability level input through the user terminal.

Then, the 5G network may determine whether to remotely control the vehicle (S2). Here, the 5G network may include a server or module for performing remote control related to autonomous driving.

In addition, the 5G network may transmit information (or signals) related to remote control to the autonomous vehicle (S3).

As described above, the information related to the remote control may be a signal directly applied to the autonomous driving vehicle, and further may further include service information required for autonomous driving. In an embodiment of the present disclosure, the autonomous driving vehicle may provide services related to autonomous driving by receiving service information such as insurance for each section selected on a driving route and information on dangerous sections through a server connected to the 5G network.

Hereinafter, in FIGS. 2 to 6 , in order to provide an insurance service applicable to each section in the autonomous driving process according to an embodiment of the present disclosure, an essential process for 5G communication between the autonomous driving vehicle and the 5G network (eg, the vehicle and the 5G network) The initial connection procedure between each other) is briefly described.

2 shows an example of an application operation of an autonomous vehicle and a 5G network in a 5G communication system.

The autonomous vehicle performs an initial access procedure with the 5G network (S20).

The initial access procedure includes a cell search for acquiring downlink (DL) synchronization, a process of acquiring system information, and the like.

Then, the autonomous vehicle performs a random access procedure with the 5G network (S21). The random access process may include a preamble transmission, random access response reception process, etc. for acquiring uplink (UL) synchronization or UL data transmission.

Then, the 5G network transmits a UL grant for scheduling transmission of specific information to the autonomous vehicle (S22). The UL grant reception includes a process of receiving time/frequency resource scheduling for transmission of UL data to a 5G network.

Then, the autonomous vehicle transmits specific information to the 5G network based on the UL grant (S23).

Then, the 5G network determines whether to remotely control the vehicle (S24).

Then, the autonomous vehicle receives a DL grant through a physical downlink control channel to receive a response to specific information from the 5G network (S25).

Then, the 5G network transmits information (or signals) related to remote control to the autonomous vehicle based on the DL grant (S26).

Meanwhile, in FIG. 2 , an example in which an initial access process and/or a random access process and a downlink grant reception process of an autonomous vehicle and 5G communication are combined through the processes S20 to S26 has been exemplarily described, but various embodiments of the present disclosure are not limited thereto.

For example, an initial access process and/or a random access process may be performed through processes S20, S22, S23, S24, and S26. Also, for example, an initial access process and/or a random access process may be performed through processes S21, S22, S23, S24, and S26. In addition, a process in which an AI operation and a downlink grant reception process are combined may be performed through S23, S24, S25, and S26.

In addition, in FIG. 2 , the autonomous vehicle operation is exemplarily described through S20 to S26, and various embodiments of the present disclosure are not limited thereto.

For example, in the autonomous vehicle operation, S20, S21, S22, and S25 may be selectively combined with S23 and S26. Also, for example, the autonomous driving vehicle operation may include S21, S22, S23, and S26. Also, for example, the autonomous driving vehicle operation may include S20, S21, S23, and S26. Also, for example, the autonomous driving vehicle operation may be configured by S22, S23, S25, and S26.

3 to 6 show an example of an autonomous driving vehicle operation using 5G communication.

Referring first to FIG. 3 , an autonomous driving vehicle including an autonomous driving module performs an initial access procedure with a 5G network based on a synchronization signal block (SSB) to obtain DL synchronization and system information ( S30 ).

Then, the autonomous vehicle performs a random access procedure with the 5G network for UL synchronization acquisition and/or UL transmission (S31).

Then, the autonomous vehicle receives a UL grant to the 5G network to transmit specific information (S32).

Then, the autonomous vehicle transmits specific information to the 5G network based on the UL grant (S33).

Then, the autonomous vehicle receives a DL grant for receiving a response to specific information from the 5G network (S34).

Then, the autonomous vehicle receives information (or signals) related to remote control from the 5G network based on the DL grant (S35).

A beam management (BM) process may be added to S30, and a beam failure recovery process related to PRACH (physical random access channel) transmission may be added to S31, and a UL grant is included in S32. A QCL relationship may be added in relation to the beam reception direction of the PDCCH, and the QCL relationship is added in relation to the beam transmission direction of a physical uplink control channel (PUCCH)/physical uplink shared channel (PUSCH) including specific information in S33. can be In addition, a QCL relationship may be added in relation to the beam reception direction of the PDCCH including the DL grant in S34.

Referring to FIG. 4 , the autonomous vehicle performs an initial access procedure with the 5G network based on the SSB to acquire DL synchronization and system information ( S40 ).

Then, the autonomous vehicle performs a random access procedure with the 5G network for UL synchronization acquisition and/or UL transmission (S41).

Then, the autonomous vehicle transmits specific information to the 5G network based on a configured grant (S42).

Then, the autonomous vehicle receives information (or signals) related to remote control from the 5G network based on the set grant (S43).

Referring to FIG. 5 , the autonomous vehicle performs an initial access procedure with the 5G network based on the SSB to obtain DL synchronization and system information ( S50 ).

Then, the autonomous vehicle performs a random access procedure with the 5G network for UL synchronization acquisition and/or UL transmission (S51).

Then, the autonomous vehicle receives a DownlinkPreemption IE from the 5G network (S52).

Then, the autonomous vehicle receives DCI format 2_1 including a preemption indication from the 5G network based on the DownlinkPreemption IE (S53).

And, the autonomous vehicle does not perform (or expect or assume) the reception of eMBB data in the resource (PRB and/or OFDM symbol) indicated by the pre-emption indication (S54).

Then, the autonomous vehicle receives a UL grant to the 5G network to transmit specific information (S55).

Then, the autonomous vehicle transmits specific information to the 5G network based on the UL grant (S56).

Then, the autonomous vehicle receives a DL grant for receiving a response to specific information from the 5G network (S57).

Then, the autonomous vehicle receives information (or signals) related to remote control from the 5G network based on the DL grant (S58).

Referring to FIG. 6 , the autonomous vehicle performs an initial access procedure with the 5G network based on the SSB to obtain DL synchronization and system information ( S60 ).

Then, the autonomous vehicle performs a random access procedure with the 5G network for UL synchronization acquisition and/or UL transmission (S61).

Then, the autonomous vehicle receives a UL grant to the 5G network to transmit specific information (S62).

The UL grant includes information on the number of repetitions for the transmission of the specific information, and the specific information is repeatedly transmitted based on the information on the number of repetitions (S63).

And, the autonomous vehicle transmits specific information to the 5G network based on the UL grant.

In addition, repeated transmission of specific information may be performed through frequency hopping, transmission of the first specific information may be transmitted in a first frequency resource, and transmission of the second specific information may be transmitted in a second frequency resource.

The specific information may be transmitted through a narrowband of 6RB (Resource Block) or 1RB (Resource Block).

Then, the autonomous vehicle receives a DL grant for receiving a response to specific information from the 5G network (S64).

Then, the autonomous vehicle receives information (or signal) related to remote control from the 5G network based on the DL grant (S65).

The above salpin 5G communication technology may be applied in combination with the methods proposed in the present specification to be described later, or may be supplemented to specify or clarify the technical characteristics of the methods proposed in the present specification.

The vehicle described in the present disclosure is connected to an external server through a communication network, and can move along a preset route without driver intervention using autonomous driving technology. The vehicle of the present disclosure may be implemented as an internal combustion engine vehicle having an engine as a power source, a hybrid vehicle having an engine and an electric motor as a power source, an electric vehicle having an electric motor as a power source, and the like.

In the following embodiments, a user may be interpreted as a driver, a passenger, or an owner of a user terminal. The user terminal may be a mobile terminal, for example, a smart phone, which is portable and capable of executing a phone call and various applications, but is not limited thereto. For example, the user terminal may be interpreted as a mobile terminal, a personal computer (PC), a notebook computer, or an autonomous vehicle system.

In an autonomous vehicle, the type and frequency of accidents can vary greatly depending on the ability to sense surrounding risk factors in real time. The route to the destination may include sections with different risk levels due to various causes, such as weather, terrain characteristics, and traffic congestion. The present disclosure guides the insurance required for each section when the user inputs a destination, and updates the insurance guide through real-time monitoring of the risk section.

At least one of an autonomous driving vehicle, a user terminal, and a server of the present disclosure, an artificial intelligence module, a drone (Unmanned Aerial Vehicle, UAV), a robot, an augmented reality (AR) device, a virtual reality, VR) and devices related to 5G services, etc.

For example, the autonomous driving vehicle may operate in connection with at least one artificial intelligence module or robot included in the vehicle.

For example, the vehicle may interact with at least one robot. The robot may be an Autonomous Mobile Robot (AMR) capable of driving by itself. The mobile robot is free to move because it can move by itself, and is provided with a plurality of sensors for avoiding obstacles while driving, so that it can run while avoiding obstacles. The mobile robot may be a flying robot (eg, a drone) having a flying device. The mobile robot may be a wheel-type robot having at least one wheel and moving through rotation of the wheel. The mobile robot may be a legged robot having at least one leg and moving using the leg.

The robot can function as a device that complements the convenience of vehicle users. For example, the robot may perform a function of moving a load loaded in a vehicle to a final destination of a user. For example, the robot may perform a function of guiding a user who got off the vehicle to a final destination. For example, the robot may perform a function of transporting a user who got out of a vehicle to a final destination.

At least one electronic device included in the vehicle may communicate with the robot through the communication device.

At least one electronic device included in the vehicle may provide the robot with data processed by the at least one electronic device included in the vehicle. For example, the at least one electronic device included in the vehicle may include at least one of object data indicating objects around the vehicle, map data, vehicle state data, vehicle location data, and driving plan data. Either one can be provided to the robot.

At least one electronic device included in the vehicle may receive, from the robot, data processed by the robot. At least one electronic device included in the vehicle may receive at least one of sensing data generated by the robot, object data, robot state data, robot position data, and movement plan data of the robot.

At least one electronic device included in the vehicle may generate a control signal based on data received from the robot. For example, the at least one electronic device included in the vehicle compares the information on the object generated by the object detection device with the information on the object generated by the robot, and generates a control signal based on the comparison result. can At least one electronic device included in the vehicle may generate a control signal to prevent interference between the movement path of the vehicle and the movement path of the robot.

At least one electronic device included in the vehicle may include a software module or a hardware module (hereinafter, referred to as an artificial intelligence module) that implements artificial intelligence (AI). At least one electronic device included in the vehicle may input acquired data to an artificial intelligence module and use data output from the artificial intelligence module.

The artificial intelligence module may perform machine learning on input data using at least one artificial neural network (ANN). The artificial intelligence module may output driving plan data through machine learning on input data.

At least one electronic device included in the vehicle may generate a control signal based on data output from the artificial intelligence module.

According to an embodiment, at least one electronic device included in a vehicle may receive data processed by artificial intelligence from an external device through a communication device. At least one electronic device included in the vehicle may generate a control signal based on data processed by artificial intelligence.

7A illustrates sensors included in a headlamp of a vehicle according to various embodiments of the present disclosure;

Referring to FIG. 7A , each of the vehicle headlamps 701 and 703 according to various embodiments includes at least two cameras 711 , 713 , 731 , 733 , at least one lidar 721 , 723 , and It may include at least one light source (741, 743).

According to various embodiments, the first headlamp 701 is at least one of a first front camera 711 , a first side camera 731 , a first lidar 721 , and at least one first light source 741 . may contain one. The first headlamp 701 may be a headlamp mounted on the right front side of the vehicle, the first front camera 711 is disposed to sense the front of the vehicle, and the first side camera 731 is, It may be arranged to sense at least a part of the front and a right direction. The first lidar 721 may be disposed to sense at least a portion of the front of the vehicle and a right direction. For example, the first front camera 711 may be disposed on the left side of the interior space area of the first headlamp 701 for sensing the front of the vehicle. The first side camera 731 may be disposed on the right side of the inner space area of the first headlamp 701 for sensing at least a part of the front of the vehicle and the right direction. The first lidar 721 may be disposed on the right side of the interior space area of the first headlamp 701 for sensing at least a part of the front of the vehicle and the right direction. In FIG. 7A , the first side camera 731 is disposed under the first lidar 721 , but this is only an example, and various embodiments of the present disclosure are not limited thereto. For example, the first side camera 731 may be disposed in any one direction of the upper, lower, left, or right side of the first lidar 721 . According to an embodiment, the first side camera 731 and the first lidar 721 may be disposed to be spaced apart from each other, or may be disposed to directly contact each other without being spaced apart. The at least one first light source 741 may be disposed in a central area of the inner space area of the first head lamp 701 . This is only an example, and various embodiments of the present disclosure are not limited thereto. For example, the at least one first light source 741 is disposed in a left area adjacent to the first front camera 711 among the internal spatial areas of the first headlamp 701 or is provided to the first side camera 731 . It may be disposed in an adjacent right area. The at least one first light source 741 may be plural. For example, the first head lamp 701 may include a plurality of light sources.

According to various embodiments, the second headlamp 703 may include at least one of a second front camera 713 , a second side camera 733 , and a second lidar 723 . The second headlamp 703 may be a headlamp mounted on the left front side of the vehicle, the second front camera 713 is arranged to sense the front of the vehicle, and the second side camera 733 is, It may be arranged to sense at least a part of the front and a left direction. The second lidar 723 may be disposed to sense at least a portion of the front of the vehicle and a left direction. For example, the second front camera 713 may be disposed on the right side of the inner space area of the second headlamp 703 for sensing the front of the vehicle. The second side camera 733 may be disposed in a left area of the interior space area of the second headlamp 703 for sensing at least a part of the front of the vehicle and the left direction. The second lidar 723 may be disposed in at least a portion of the front of the vehicle and in a left area of the interior space area of the second headlamp 703 for left direction sensing. In FIG. 7A , the second side camera 733 is disposed under the second lidar 723 , but this is only an example, and various embodiments of the present disclosure are not limited thereto. For example, the second side camera 733 may be disposed in any one direction of the top, bottom, left, or right side of the second lidar 723 . According to an embodiment, the second side camera 733 and the second lidar 723 may be disposed to be spaced apart from each other, or may be disposed to directly contact each other without being spaced apart. The at least one second light source 743 may be disposed in a central area of the inner space area of the second head lamp 703 . This is only an example, and various embodiments of the present disclosure are not limited thereto. For example, the at least one second light source 743 may be disposed in a left area adjacent to the second front camera 713 in the inner space area of the second headlamp 703 or to the second side camera 733 . It may be disposed in an adjacent right area. The at least one second light source 743 may be plural. For example, the second head lamp 703 may include a plurality of light sources.

7B illustrates a field of view (FOV) of sensors in a headlamp according to various embodiments. The headlamps mounted on the vehicle of FIG. 7B may be the headlamps 701 and 703 illustrated in FIG. 7A .

Referring to FIG. 7B , the vehicle uses the first front camera 711 included in the first headlamp 701 and the second front camera 713 included in the second headlamp 703 to be positioned in front of the object. can be sensed. A field of view of each of the first front camera 711 and the second front camera 713 may be, for example, about 40 degrees.

The vehicle senses an object located in the front and/or side by using the first lateral camera 731 included in the first headlamp 701 and the second lateral camera 733 included in the second headlamp 703 . can do. The field of view of each of the first side camera 731 and the second side camera 733 may be, for example, about 140 degrees.

The vehicle senses an object located in the front and/or side by using the first lidar 721 included in the first headlamp 701 and the second lidar 723 included in the second headlamp 703 . can do. A field of view of each of the first lidar 721 and the second lidar 723 may be, for example, about 120 degrees.

The above-described viewing angles are merely examples, and various embodiments of the present disclosure are not limited thereto. For example, the viewing angles of the sensors 711 , 731 , 721 , 723 , 731 , and 733 included in the first headlamp 701 and the second headlamp 703 may be set to different angles by a designer. there is.

In addition, in the above-described FIGS. 7A and 7B and the embodiments to be described later, it is assumed that one lidar is included in each of the two headlamps 701 and 703 located at the front of the vehicle, but the present disclosure Various embodiments of , are not limited thereto. For example, various embodiments of the present disclosure may be applied in the same manner even when a plurality of lidars are included in each of the headlamps 701 and 703 . In addition, the at least two cameras 711 , 713 , 731 , 733 , at least one lidar 721 , 723 , and at least one light source 741 , 743 may be applied to a tail lamp, a fog lamp, etc. in the same manner. there is.

Although not shown in FIGS. 7A and 7B , the above-described vehicle may further include at least one rear side camera. For example, it may further include a first rear side camera sensing at least a part of the rear of the vehicle and a right direction, and a second rear side camera sensing at least a part of the rear side of the vehicle and a left direction. The first rear side camera may be disposed on a right rear lamp of the vehicle, and the second rear side camera may be disposed on a left rear lamp of the vehicle.

Hereinafter, for convenience of description, the first front camera 711 shown in FIGS. 7A and 7B is referred to as a right front camera, the second front camera 713 is referred to as a left front camera, and the first side The camera 731 may be referred to as a right side camera, and the second side camera 733 may be referred to as a left side camera.

8 is a control block diagram of a vehicle according to an embodiment of the present disclosure.

Referring to FIG. 8 , the vehicle includes a user interface device 800 , an object detection device 810 , a communication device 820 , a driving manipulation device 830 , a main ECU 840 , a driving control device 850 , and an autonomous vehicle. It may include a driving device 860 , a sensing unit 870 , and a location data generating device 880 . The object detecting device 810 , the communication device 820 , the driving manipulation device 830 , the main ECU 840 , the driving control device 850 , the autonomous driving device 860 , the sensing unit 870 , and the location data generating device 880 may be implemented as electronic devices that each generate electrical signals and exchange electrical signals with each other.

The user interface device 800 is a device for communication between a vehicle and a user. The user interface device 800 may receive a user input and provide information generated in the vehicle to the user. The vehicle may implement a user interface (UI) or a user experience (UX) through the user interface device 800 . The user interface device 800 may include an input device, an output device, and a user monitoring device.

The object detecting apparatus 810 may generate information about an object outside the vehicle. The information about the object may include at least one of information on the existence of the object, location information of the object, distance information between the vehicle and the object, and relative speed information between the vehicle and the object. The object detecting apparatus 810 may detect an object outside the vehicle. The object detecting apparatus 810 may include at least one sensor capable of detecting an object outside the vehicle. The object detecting apparatus 810 may include at least one of a camera, a radar, a lidar, an ultrasonic sensor, and an infrared sensor. The object detecting apparatus 810 may provide data on an object generated based on a sensing signal generated by a sensor to at least one electronic device included in the vehicle.

The camera may generate information about an object outside the vehicle by using the image. The camera may include at least one lens, at least one image sensor, and at least one image signal processor. The image signal processor may be electrically connected to the image sensor, process a received signal, and generate data about the object based on the processed signal. The camera may be at least one of a mono camera, a stereo camera, and an AVM (Around View Monitoring) camera. The camera may obtain position information of the object, distance information from the object, or relative speed information with the object by using various image processing algorithms. For example, the camera may acquire distance information and relative velocity information from an object based on a change in the size of the object over time from the acquired image. For example, the camera may acquire distance information and relative speed information with respect to an object through a pinhole model, road surface profiling, or the like. For example, the camera may acquire distance information and relative velocity information from an object based on disparity information in a stereo image obtained from the stereo camera.

The camera may be mounted at a position where a field of view (FOV) can be secured in the vehicle in order to photograph the outside of the vehicle. The camera may be disposed adjacent to the front windshield in the interior of the vehicle to acquire an image of the front of the vehicle. The camera may be placed around the front bumper or radiator grill. The camera may be disposed adjacent to the rear glass in the interior of the vehicle to acquire an image of the rear of the vehicle. The camera may be placed around the rear bumper, trunk or tailgate. The camera may be disposed adjacent to at least one of the side windows in the interior of the vehicle in order to acquire an image of the side of the vehicle. Alternatively, the camera may be disposed around a side mirror, a fender or a door.

The radar may generate information about an object outside the vehicle using radio waves. The radar may include an electromagnetic wave transmitter, an electromagnetic wave receiver, and at least one processor. The at least one processor may be electrically connected to the electromagnetic wave transmitter and the electromagnetic wave receiver, process the received signal, and generate data for the object based on the processed signal. The radar may be implemented in a pulse radar method or a continuous wave radar method in terms of a radio wave emission principle. The radar may be implemented by a frequency modulated continuous wave (FMCW) method or a frequency shift keying (FSK) method according to a signal waveform among continuous wave radar methods. The radar detects an object based on an electromagnetic wave, a time of flight (TOF) method or a phase-shift method, and detects the position of the detected object, the distance to the detected object, and the relative speed. can The radar may be placed at a suitable location outside of the vehicle to detect objects located in front, rear or side of the vehicle.

The lidar may generate information about an object outside the vehicle by using the laser light. The lidar may include a light transmitter, a light receiver, and at least one processor. It may be electrically connected to the light transmitter and the light receiver, process a received signal, and generate data for an object based on the processed signal. The lidar may be implemented in a time of flight (TOF) method or a phase-shift method. Lidar can be implemented as driven or non-driven. When implemented as a driving type, the lidar is rotated by a motor and can detect objects around the vehicle. When implemented as a non-driven type, the lidar may detect an object located within a predetermined range with respect to the vehicle by light steering. Vehicle 100 may include a plurality of non-driven lidar. LiDAR detects an object based on a time of flight (TOF) method or a phase-shift method with a laser light medium, and calculates the position of the detected object, the distance to the detected object, and the relative speed. can be detected. The lidar may be placed at a suitable location outside of the vehicle to detect an object located in front, rear or side of the vehicle.

The communication apparatus 820 may exchange signals with a device located outside the vehicle. The communication device 820 may exchange signals with at least one of an infrastructure (eg, a server, a broadcasting station), another vehicle, and a terminal. The communication device 820 may include at least one of a transmission antenna, a reception antenna, a radio frequency (RF) circuit capable of implementing various communication protocols, and an RF element to perform communication.

For example, the communication device may exchange signals with an external device based on V2X technology. For example, the V2X technology may include LTE-based sidelink communication and/or NR-based sidelink communication. In addition, the communication device can exchange information with objects such as other vehicles, mobile devices, and roads based on the 5G network. The contents related to V2X will be described later.

For example, the communication device is based on IEEE 80211p PHY/MAC layer technology and IEEE 1609 Network/Transport layer technology-based Dedicated Short Range Communications (DSRC) technology or WAVE (Wireless Access in Vehicular Environment), SAEJ2735, SAE J2945 standards. Signals can be exchanged with external devices. DSRC (or WAVE standard) technology is a communication standard prepared to provide an Intelligent Transport System (ITS) service through short-distance dedicated communication between in-vehicle devices or between roadside devices and in-vehicle devices. The DSRC technology may use a frequency of the 59 GHz band and may be a communication method having a data transmission rate of 3 Mbps to 27 Mbps. The IEEE 80211p technology may be combined with the IEEE 1609 technology to support DSRC technology (or WAVE standard).

The communication apparatus of the present disclosure may exchange a signal with an external device using only one of the V2X technology or the DSRC technology. Alternatively, the communication apparatus of the present disclosure may exchange a signal with an external device by hybridizing the V2X technology and the DSRC technology. The V2X standard was created through IEEE (IEEE 80211p, IEEE 1609), a standardization organization in the electrical and electronic field, and SAE (SAE J2735, SAE J2945, etc.), a group of automotive engineers. in charge In particular, with respect to the message standard, SAE established standards for defining the message standard for V2X communication.

The driving operation device 830 is a device that receives a user input for driving. In the manual mode, the vehicle may be driven based on a signal provided by the driving manipulation device 830 . The driving manipulation device 830 may include a steering input device (eg, a steering wheel), an accelerator input device (eg, an accelerator pedal), and a brake input device (eg, a brake pedal).

The main ECU 840 may control the overall operation of at least one electronic device included in the vehicle.

The drive control device 850 is a device that electrically controls various vehicle drive devices in the vehicle. The drive control device 850 may include a power train drive control device, a chassis drive control device, a door/window drive control device, a safety device drive control device, a lamp drive control device, and an air conditioning drive control device. The power train drive control device may include a power source drive control device and a transmission drive control device. The chassis drive control device may include a steering drive control device, a brake drive control device, and a suspension drive control device. Meanwhile, the safety device drive control device may include a safety belt drive control device for seat belt control.

The drive control device 850 includes at least one electronic control device (eg, a control ECU (Electronic Control Unit)).

The driving control device 850 may control the vehicle driving device based on a signal received from the autonomous driving device 860 . For example, the drive control device 850 may control a power train, a steering device, and a brake device based on a signal received from the autonomous driving device 860 .

The autonomous driving device 860 may generate a route for autonomous driving based on the obtained data. The autonomous driving device 860 may generate a driving plan for driving along the generated path. The autonomous driving device 860 may generate a signal for controlling the movement of the vehicle according to the driving plan. The autonomous driving device 860 may provide the generated signal to the driving control device 850 .

The autonomous driving device 860 may implement at least one Advanced Drive Assistance System (ADAS) function. ADAS includes Adaptive Cruise Control (ACC), Autonomous Emergency Braking (AEB), Forward Collision Warning (FCW), and Lane Keeping Assist (LKA). ), Lane Change Assist (LCA), Target Following Assist (TFA), Blind Spot Detection (BSD), Adaptive Headlight System (AHS) , Auto Parking System (APS), Pedestrian Collision Warning System (PD Collision Warning System), Traffic Sign Recognition (TSR), Traffic Sign Assist (TSA), Night Vision System At least one of a Night Vision (NV), a Driver Status Monitoring (DSM), and a Traffic Jam Assist (TJA) may be implemented.

The autonomous driving device 860 may perform a switching operation from the autonomous driving mode to the manual driving mode or a switching operation from the manual driving mode to the autonomous driving mode. For example, the autonomous driving device 860 may change the mode of the vehicle from the autonomous driving mode to the manual driving mode or to switch from the manual driving mode to the autonomous driving mode based on the signal received from the user interface device 800 . can

The sensing unit 870 may sense the state of the vehicle. The sensing unit 870 may include an inertial measurement unit (IMU) sensor, a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight sensor, a heading sensor, a position module, and a vehicle. It may include at least one of a forward/reverse sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor, a temperature sensor, a humidity sensor, an ultrasonic sensor, an illuminance sensor, and a pedal position sensor. Meanwhile, an inertial measurement unit (IMU) sensor may include at least one of an acceleration sensor, a gyro sensor, and a magnetic sensor.

The sensing unit 870 may generate state data of the vehicle based on a signal generated by at least one sensor. The vehicle state data may be information generated based on data sensed by various sensors provided inside the vehicle. The sensing unit 870 may include vehicle attitude data, vehicle motion data, vehicle yaw data, vehicle roll data, vehicle pitch data, vehicle collision data, vehicle direction data, vehicle angle data, and vehicle speed. data, vehicle acceleration data, vehicle inclination data, vehicle forward/reverse data, vehicle weight data, battery data, fuel data, tire pressure data, vehicle interior temperature data, vehicle interior humidity data, steering wheel rotation angle data, vehicle exterior illumination Data, pressure data applied to the accelerator pedal, pressure data applied to the brake pedal, and the like may be generated.

The location data generating apparatus 880 may generate location data of the vehicle. The location data generating apparatus 880 may include at least one of a Global Positioning System (GPS) and a Differential Global Positioning System (DGPS). The location data generating apparatus 880 may generate location data of the vehicle based on a signal generated by at least one of GPS and DGPS. According to an embodiment, the location data generating apparatus 880 may correct location data based on at least one of an Inertial Measurement Unit (IMU) of the sensing unit 870 and a camera of the object detecting apparatus 810 . The location data generating device 880 may be referred to as a Global Navigation Satellite System (GNSS).

The vehicle, although not shown, may include an internal communication system. A plurality of electronic devices included in the vehicle may exchange signals through an internal communication system. Signals may contain data. The internal communication system may use at least one communication protocol (eg, CAN, LIN, FlexRay, MOST, Ethernet).

9 is a diagram conceptually illustrating an electronic device 900 according to various embodiments of the present disclosure. 10A to 10C are diagrams for explaining an operation of generating a grid map for a road surface in the electronic device 900 according to various embodiments of the present disclosure. The electronic device 900 illustrated in FIG. 9 may be defined as an autonomous driving device as at least some of the components included in the vehicle of FIGS. 7A, 7B, and 8 . The configuration of the electronic device 900 illustrated in FIG. 9 is an example, and each component may be composed of one chip, component, or electronic circuit, or a combination of chips, components, or electronic circuit. According to another embodiment, some of the components shown in FIG. 9 may be divided into a plurality of components and configured as different chips or components or electronic circuits, and some components may be combined to form one chip, component, or It may consist of an electronic circuit. According to another embodiment, some of the components shown in FIG. 9 may be omitted or other components not shown in FIG. 9 may be added. According to an embodiment, some of the components shown in FIG. 9 are components included in the vehicle rather than included in the electronic device 900 , and at least one other component included in the electronic device 900 . may be electrically connected to the element.

Referring to FIG. 9 , an electronic device 900 according to various embodiments includes a first sensor 912 , a second sensor 914 , a third sensor 920 , a memory 930 , a communication circuit 940 , and a processor. 950 . According to an embodiment, the first sensor 912 and the second sensor 914 may be included in the lamp 910 of the vehicle, but the present disclosure is not limited thereto. For example, only one of the first sensor 912 or the second sensor 914 is included in the lamp 910 or the third sensor 920 along with the first sensor 912 and the second sensor 914 is also included in the lamp ( 910) may be included. Also, the first sensor 912 and the second sensor 914 may be disposed outside the lamp 910 .

According to various embodiments, the lamp 910 illuminates the front of the driving path under the control of the processor 950 . The lamp 910 may be the headlamps 701 and 703 of the vehicle shown in FIGS. 7A and 7B . According to an embodiment, the lamp 910 may include at least one first sensor 912 and at least one second sensor 914 . In addition, the lamp 910 may additionally or alternatively include at least one light source, a reflector that reflects light emitted from the light source, a component within the lamp 910 (eg, at least one sensor 912 , 914 , 920 ), or at least one at least one driving body (eg, a motor) configured to rotate at least one of the light sources) in a horizontal direction and/or a vertical direction, and a protective cover for protecting at least one component of the lamp 910 , etc. You may.

According to an embodiment, the first sensor 912 may include at least one camera for acquiring at least one image of the outside of the vehicle (or the electronic device 900 ). For example, the at least one camera may include at least one of front cameras 711 and 713 and side cameras 731 and 733 .

According to an embodiment, the second sensor 914 may include at least one lidar configured to detect an object by emitting a laser beam. For example, the at least one lidar may include at least one of a first lidar 721 or a second lidar 723 as shown in FIGS. 7A and/or 7B . For example, the at least one lidar emits a laser beam in a direction requested by the processor 950 when the vehicle is driving, and based on the laser beam reflected and received by the object, it is possible to generate data about the object. have.

According to various embodiments, the third sensor 920 may include at least one sensor (eg, the sensor unit 870) for collecting driving information of the vehicle. The driving information may include at least one of speed, acceleration, and wheel rotation angle, but is not limited thereto. In addition, the driving information may include information related to the type of road on which the vehicle is traveling (eg, a highway, a general national road, a local road, etc.), but is not limited thereto.

According to various embodiments, the memory 930 includes at least one component (eg, a first sensor 912 , a second sensor 914 , a third sensor 920 , a communication circuit 940 , or a processor 950 ). ) can store various data used by According to an embodiment, the memory 930 may store high-precision map data.

According to various embodiments, the communication circuit 940 may communicate with at least one external device based on V2X technology or 5G network. According to an embodiment, the communication circuit 940 may be the communication device 820 of FIG. 8 .

According to various embodiments, the processor 950 may be configured to control the overall operation of the electronic device 900 . For example, the processor 950 may perform a control operation related to autonomous driving of the vehicle. According to an embodiment, the processor 950 executes software (eg, a program) stored in the memory 930 , and a component (eg, a lamp 910 , a first sensor 912 ) connected to the processor 950 . , the second sensor 914 , the third sensor 920 , the memory 930 , or the communication circuit 940 ). For example, the processor 950 may include a processor having an arithmetic processing function. For example, the processor 950 may include an arithmetic processing unit such as a central processing unit (CPU), a micro computer unit (MCU), or a graphics processing unit (GPU). However, the present invention is not limited thereto.

According to an embodiment, the processor 950 may identify the free area based on first sensing data (eg, image data of the front of the vehicle) acquired through the first sensor 912 . The free area may mean an area (or space) in which the vehicle can move among areas around the vehicle (or the electronic device 900 ). For example, the processor 950 may detect an object around the vehicle, and based on this, identify a free area in which the vehicle can move based on the current location of the vehicle. For example, the processor 950 is free based on at least one of vehicle specifications (eg, overall length, overall width, total height, wheelbase, wheelbase, weight, etc.), driving information of the vehicle, road information, or object information (eg, object type). area can be checked.

According to an embodiment, in response to the confirmation of the free area, the processor 950 may acquire second sensing data for the free area based on the second sensor 914 . The second sensing data may be data collected by being reflected by the free area among data (eg, point cloud) collected by the operation of the second sensor 914 . For example, after mapping the first sensing data and the second sensing data, the processor 950 may extract only the second sensing data corresponding to the free space. At this time, in order to map the first sensing data and the second sensing data, the processor 950 includes internal parameters of the first sensor 912 (eg, focal length, principal point, skew coefficient). coefficient) and external parameters of the first sensor 912 (eg, the installation height and direction of the first sensor 912), and the like to perform camera calibration. However, this is only an example, and the embodiment of the present disclosure is not limited thereto, and various known methods may be applied to the mapping of sensing data.

According to an embodiment, in response to the acquisition of the second sensing data, the processor 950 generates map information (eg, a grid map) representing the height of a road surface (eg, a free area) based on the second sensing data. can do. For example, the processor 950 may generate a plurality of grids having a specified size including the free area (or corresponding to the free area), and group the second sensing data corresponding to each grid. Also, the processor 950 may use the grouped second sensing data as height information for the corresponding grid. For example, as shown in FIGS. 10A to 10C , the processor 950 provides an average value in a first direction (eg, X-axis) and an average value in a second direction (eg, Y-axis) for the distributed second sensing data. can be calculated and used as height information (or height variation information) of the corresponding grid. According to an embodiment, as shown in FIG. 10A , when there is no abnormal section in the free area 1007 between the vehicle 1003 and the other vehicle 1005 ( 1001 ), the processor 950 changes the height. A grid map 1009 composed of a grid indicating that there is no may be generated ( 1000 ). According to another embodiment, as shown in FIG. 10B , when an abnormal section such as the speed bump 1015 exists in the free area 1007 ( 1011 ), the processor 950 generates a grid corresponding to the edge of the speed bump. A grid map 1019 representing a height increase 1017 can be generated (1010) for the fields. According to another embodiment, as shown in FIG. 10C , when an abnormal section such as a pothole 1025 exists in the free region 1007 ( 1021 ), the processor 950 corresponds to the edge of the pothole. A grid map 1029 representing a height reduction 1027 may be generated (1020) for the grids. Among the numbers described in each object shown in FIGS. 10A to 10C , the number written on the upper side may be height information for the first direction, and the number written on the lower side may be height information for the second direction.

According to an embodiment, in response to the generation of map information, the processor 950 may predict a road surface condition based on a pattern of the map information. The pattern may be a pattern formed by grids indicating a change in height among map information. For example, the processor 950 may identify an abnormal section in which a height change exists in the free area based on a pattern of grids indicating a height change among the grids. For example, the processor 950 may predict the state of the road surface based on the pattern of the grid map and the height change value.

According to an embodiment, in response to predicting the road surface condition, the processor 950 may control the driving of the vehicle based on the road surface condition. For example, based on the road surface condition, the processor 950 may provide a warning to the driver when a vehicle departs from an intelligent cruise control system, a highway driving assist system, or a driving lane, or the lane in which the vehicle was originally driving, based on the road surface condition. Various driving functions may be provided, such as a lane keeping assist system for returning the vehicle to the vehicle. As another example, the processor 950 may adjust the light output method of the lamp 910 based on the road surface condition. For example, the processor 950 may adjust at least one of the intensity of the light source and/or the irradiation direction of the light source so that light is concentrated in the abnormal section. As another example, the processor 950 may notify the prediction result of the road surface condition (eg, a pothole is detected in the front right portion) using an output device such as a display or a speaker.

According to various embodiments, at least a part of the operation of the electronic device 900 described above may be determined using a machine learning algorithm or determined based on generated information, and the processor 950 is a component of the electronic device 900 . can be controlled to perform the determined operation. For example, the processor 950 may predict a free area and a road surface condition through machine learning.

11 is a flowchart 1100 illustrating a method for predicting a road surface condition in the electronic device 900 according to embodiments of the present disclosure. In the following embodiment, each operation may be performed sequentially, but is not necessarily performed sequentially. In addition, the following operations may be performed by the processor 950 of the electronic device 900 or implemented as instructions executable by the processor 950 .

9 and 11 , the electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments determines a free area for the driving direction of the vehicle based on the first sensing data in operation S1110 . can be checked According to an embodiment, the first sensed data may be at least one image of the outside of the vehicle (or the electronic device 900 ). For example, the electronic device 900 may acquire first sensing data of the front of the vehicle through at least one first sensor 912 (eg, a camera) included in the lamp 910 . However, this is only an example, and the embodiment of the present disclosure is not limited thereto. For example, the electronic device 900 may identify a free area for the side or rear of the vehicle. According to an embodiment, the electronic device 900 may identify the free area by analyzing the first sensed data using a machine learning algorithm.

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may acquire second sensing data for the free area based on the second sensor 914 in operation S1120 . The free area may mean an area (or space) in which the vehicle can move among areas around the vehicle (or the electronic device 900 ). Also, the second sensing data may be data (eg, point cloud) collected by the operation of the second sensor 914 . According to an embodiment, after mapping the first sensing data and the second sensing data, the electronic device 900 may extract only the second sensing data corresponding to the free space.

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may generate map information representing the height of the road surface based on the second sensing data in operation S1130 . The map information may be a grid map including height information of an object detected using the second sensor 914 . According to an embodiment, the electronic device 900 may express the height of the road surface based on the average value of the collected second sensing data, as will be described later with reference to FIG. 12 .

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may predict a road surface condition based on a pattern of map information in operation S1140 . The road surface condition may include a state in which a speed bump is installed, a state in which a pothole is generated, a state in which a pothole is generated, a state in which an ice film is formed, and the like. According to an embodiment, the electronic device 900 may predict the road surface condition based on a pattern of at least one grid indicating a height change among grids included in the map information. For example, the electronic device 900 may predict the state of the road surface based on a pattern of a grid map and a height change value. For example, if the electronic device 900 identifies a pattern having a regular area of a certain level in which the height is increased to a certain level relative to its surroundings, it may predict that the speed bump is installed on the front road surface. Also, if the electronic device 900 identifies a pattern having an irregular area of a certain level in which the height is reduced to a certain level relative to its surroundings, the electronic device 900 may predict the presence of a pothole on the front road surface. Also, the electronic device 900 may determine that the front road surface is wet when checking the pattern of the transparent area in which a certain level of area is recognized as a distance from the road surface.

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may control the vehicle based on the road surface condition in operation S1150 . According to an embodiment, the electronic device 900 may control the driving of the vehicle based on the road surface condition. For example, the electronic device 900 may use an intelligent cruise control system, a highway driving assist system, or a lane keeping system to warn the driver or return to the original driving lane when the vehicle departs from the driving lane. Various driving functions such as a lane keeping assist system may be provided. According to another embodiment, the electronic device 900 may adjust the light output method of the lamp 910 based on the rough surface state. According to another embodiment, the electronic device 900 may notify the prediction result of the road surface condition (eg, a pothole is detected in the front right portion) using an output device such as a display or a speaker.

The electronic device 900 according to various embodiments may omit at least one of the above-described operations. According to an embodiment, the electronic device 900 may selectively perform the operation S1150 of controlling the vehicle based on the road surface condition.

12 is a flowchart 1200 illustrating a method of generating map information representing the height of a road surface in the electronic device 900 according to embodiments of the present disclosure. The operations of FIG. 12 described below may represent various embodiments of operation S1130 of FIG. 11 . In addition, in the following embodiment, each operation is not necessarily performed sequentially, and may be performed by the processor 950 of the electronic device 900 or implemented as instructions executable by the processor 950 .

Referring to FIGS. 9 and 12 , the electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may generate a plurality of grids including free regions in operation S1210 . According to an embodiment, the electronic device 900 may determine the horizontal and vertical sizes of the free area and generate a plurality of grids having a predetermined size. For example, the size of each grid may be the same or different from each other.

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may classify sensing data corresponding to each grid in operation S1220 . According to an embodiment, the electronic device 900 maps the generated grid with the second sensing data reflected by the free area among data (eg, point cloud) collected by the operation of the second sensor 914 and collected. Thus, the second sensing data corresponding to each grid may be grouped and classified.

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may measure the height of the road surface corresponding to each grid based on the second sensing data classified in operation S1230 . According to an embodiment, the electronic device 900 checks the distribution of the second sensing data corresponding to each lattice, and the average value of the distributed second sensing data in the first direction (eg, the X axis) and the second direction (Example: Y axis) An average value can be calculated, and this can be used as height information (or height variation information) of the corresponding grid. For example, the electronic device 900 may measure a changing vector value between gratings as the height of the road surface based on the scan direction of the second sensor 914 .

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may generate a grid map in which the height of the road surface corresponding to each grid is expressed in operation S1230 . According to an embodiment, the electronic device 900 may generate a grid map according to road surface conditions as shown in FIGS. 10A to 10C .

13 is a flowchart 1300 illustrating a method of generating a grid for a free region in the electronic device 900 according to embodiments of the present disclosure. The operations of FIG. 13 described below may represent various embodiments of the operation S1210 of FIG. 12 . In addition, in the following embodiment, each operation is not necessarily performed sequentially, and may be performed by the processor 950 of the electronic device 900 or implemented as instructions executable by the processor 950 .

9 and 13 , the electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may acquire driving information related to a vehicle in operation S1310 . According to an embodiment, the driving information may include at least one of a vehicle speed, an acceleration, and a wheel rotation angle. For example, the electronic device 900 may acquire driving information based on the at least one third sensor 920 . According to another embodiment, the driving information may be information related to a road on which the vehicle is driving. For example, the electronic device 900 may acquire driving information from a nearby external device based on the at least one communication circuit 940 . As another example, the electronic device 900 may acquire driving information based on high-precision map data stored in the electronic device 900 (eg, the memory 930 ).

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may determine whether the driving condition obtained in operation S1320 satisfies a specified condition. According to an embodiment, the specified condition may be associated with a situation in which precise prediction of the road surface condition is required. For example, the electronic device 900 may determine that a general prediction requiring a general process is necessary for a low-speed driving environment (or a driving environment on a general national road or a local road) of a reference speed or less. Also, the electronic device 900 may determine that a high-speed driving environment (or a driving environment on a highway) with a reference speed or higher is required for precise prediction that requires a relatively high process.

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments of the present disclosure sets the first resolution in operation S1330 in response to a driving condition that satisfies a specified condition, that is, a situation in which general prediction is required. Branches can create grids.

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments of the present disclosure responds to a driving condition that does not satisfy a specified condition, that is, a situation in which precise prediction is required, in operation S1340 , the first resolution It is possible to create a grating having a higher second resolution. The electronic device 900 may improve the prediction performance of the abnormal section by setting a region having a relatively small size as the prediction range by increasing the grid resolution for a situation in which precise prediction is required. occurrence can be prevented.

14 is a flowchart 1400 illustrating a method of controlling a vehicle based on a road surface condition in the electronic device 900 according to embodiments of the present disclosure. The operations of FIG. 14 described below may represent various embodiments of the operation S1150 of FIG. 11 . In addition, in the following embodiment, each operation is not necessarily performed sequentially, and may be performed by the processor 950 of the electronic device 900 or implemented as instructions executable by the processor 950 .

9 and 14 , the electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may identify an abnormal section of the road surface in operation S1410 . The abnormal section is a section of the road surface in which factors affecting the driving of the vehicle exist, and it may include at least one of a wet section of the road surface, a section where an ice film is generated, a section where a speed bump is installed, and a section in which a pothole is generated. , but not limited thereto. According to an embodiment, the electronic device 900 may identify at least one of the location, size, and shape of the abnormal section based on the pattern of map information.

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may control the ramp 910 based on an abnormal section in operation S1420 . According to an embodiment, the electronic device 900 may adjust at least one of the intensity of the light source and/or the irradiation direction of the light source so that light is concentrated in the abnormal section.

15 is a flowchart 1500 illustrating a method of analyzing an abnormal section of a road surface in the electronic device 900 according to embodiments of the present disclosure. The operations of FIG. 15 described below may represent various embodiments of operation S1420 of FIG. 14 . In addition, in the following embodiment, each operation is not necessarily performed sequentially, and may be performed by the processor 950 of the electronic device 900 or implemented as instructions executable by the processor 950 .

9 and 15 , the electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may acquire an additional image of an abnormal section in operation S1510 . The additional image may be acquired in a situation where the ramp 910 is controlled based on the abnormal section. According to an embodiment, the electronic device 900 displays an additional image with increased illuminance for an abnormal section using the first sensor 912 included in the lamp 910 or an additional sensor disposed outside the lamp 910 . can be obtained

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may analyze an abnormal section based on the previous image and the additional image in operation S1520 . According to an embodiment of the present disclosure, the electronic device 900 may check a change in brightness for an abnormal section by comparing the previous image with the additional image, and perform a detailed analysis on the abnormal section based on the change in brightness. For example, comparing the previous image with the additional image, the degree of increase in brightness of an abnormal portion recessed from the road surface, such as a pothole, may be lower than that of a normal section of the road surface. Based on the difference, the type, shape, size, etc. of the abnormal section can be precisely analyzed.

The electronic device 900 (eg, the autonomous driving device 860 ) according to various embodiments may notify the analysis result of the abnormal section using an output device such as a display or a speaker in operation S1520 .

According to various embodiments, the autonomous driving device (eg, the electronic device 900 ) includes a first sensor (eg, the first sensor 912 ) included in a lamp (eg, the lamp 910 ) of the autonomous driving vehicle; a second sensor (eg, a second sensor 914), the first sensor, and a processor (eg, a processor 950) electrically connected to the second sensor, wherein the processor is configured to: A free area in front of which the vehicle can move is checked based on first sensing data obtained through a first sensor, and second sensing data for the free area is obtained based on the second sensor, and the second sensing It is possible to generate map information expressing the height of the road surface based on the data, and control to predict the road surface condition based on the pattern of the map information.

According to various embodiments, the first sensor may include a camera, and the second sensor may include a lidar.

According to various embodiments, the processor generates, by the autonomous vehicle, a plurality of grids including the free area, classifies the second sensing data to correspond to each of the generated grids, and 2 Based on the sensed data, the height of the road surface corresponding to each of the grids is measured, and a grid map in which the height corresponding to each of the grids is expressed may be generated.

According to various embodiments, the autonomous driving device is at least one of a third sensor (eg, the third sensor 920) and a communication circuit (eg, the communication circuit 940) for collecting driving information of the autonomous vehicle further comprising: in response to the autonomous vehicle acquiring driving information that satisfies a specified first condition, generates a grid having a first resolution, and generates driving information that satisfies a specified second condition In response to obtaining , it is possible to control to generate a grating having a second resolution different from the first resolution. According to an embodiment, the communication circuit may support at least one of V2X and 5G communication methods.

According to various embodiments of the present disclosure, the autonomous driving apparatus further includes at least one light source included in the lamp, and the processor is configured to: in response to the autonomous driving vehicle predicting a road surface condition including an abnormal section, the It is possible to control to adjust the light emission of the light source based on the abnormal section.

According to various embodiments of the present disclosure, the processor may control the autonomous vehicle to adjust at least one of an irradiation direction of the light source and an irradiation intensity of the light source based on the abnormal section.

According to various embodiments, the processor may be configured such that the autonomous vehicle acquires third sensing data for the abnormal section by using the first sensor, and based on the first sensing data and the third sensing data, It can be controlled to analyze the abnormal section.

According to various embodiments, the processor may control the autonomous driving vehicle to notify the analysis result of the abnormal section.

According to various embodiments, the processor maps the second sensing data to the first sensing data, and the processor maps the second sensing data to the first sensing data, and the second sensing data corresponding to the free area among the mapped second sensing data can be obtained and controlled to generate the map information.

According to various embodiments, the method of operating the autonomous driving device (eg, the electronic device 900 ) includes a first sensor (eg, the first sensor 912 ) included in a lamp (eg, the lamp 910 ) of the autonomous driving vehicle. ) to determine the front free area based on the first sensing data, and to obtain the second sensing data for the free area based on the second sensor (eg, the second sensor 914) included in the lamp operation, generating map information representing the height of the road surface based on the second sensing data, and predicting a road surface condition based on a pattern of the map information.

According to various embodiments, the first sensor may include a camera, and the second sensor may include a lidar.

According to various embodiments, the generating of the map information includes generating a plurality of grids including the free area, classifying the second sensing data to correspond to each of the generated grids, and the classified first 2 Based on the sensing data, the method may include measuring a height of a road surface corresponding to each of the grids and generating a grid map in which a height corresponding to each of the grids is expressed.

According to various embodiments, the generating of the grid includes generating a grid having a first resolution in response to collecting driving information for the autonomous vehicle and acquiring driving information satisfying a specified first condition. and generating a grid having a second resolution different from the first resolution in response to obtaining driving information that satisfies the specified second condition.

According to various embodiments, the driving information may include a third sensor (eg, a third sensor 920 ) different from the first sensor and the second sensor, or a communication circuit (eg, a communication circuit 940 ) of the autonomous driving device. )) is collected through at least one, and the communication circuit may support at least one of V2X, or 5G communication method.

According to various embodiments of the present disclosure, the operating method of the autonomous driving apparatus may include, in response to a road surface condition including an abnormal section being predicted, controlling the light emission of the lamp based on the abnormal section.

According to various embodiments, the operation of controlling the light emission of the lamp may include an operation of controlling at least one of an irradiation direction of the lamp and an irradiation intensity of the lamp based on the abnormal section.

According to various embodiments, the operation of controlling the light emission of the lamp is based on the operation of obtaining third sensing data for the abnormal section using the first sensor and the first sensing data and the third sensing data. , analyzing the abnormal section.

According to various embodiments, the operation of analyzing the abnormal section may include an operation of notifying an analysis result of the abnormal section.

According to various embodiments, the acquiring of the second sensing data may include mapping the second sensing data to the first sensing data and the second sensing corresponding to the free region among the mapped second sensing data. It may include an operation of acquiring data.

The method of operating the electronic device 900 according to embodiments of the present disclosure may be implemented as instructions that are stored in a computer-readable storage medium and executed by a processor (eg, the processor 950 of FIG. 9 ).

A storage medium, whether directly and/or indirectly, in a raw, formatted, organized or any other accessible state, may include a relational database, a non-relational database, an in-memory database, Alternatively, it may include a database, including a distributed one, such as any other suitable database capable of storing data and allowing access to such data through a storage controller. In addition, the storage medium includes a primary storage device (storage), a secondary storage device, a tertiary storage device, an offline storage device, a volatile storage device, a non-volatile storage device, a semiconductor storage device, a magnetic storage device, an optical storage device, a flash. It may include any type of storage device, such as a storage device, a hard disk drive storage device, a floppy disk drive, magnetic tape, or other suitable data storage medium.

Although the present disclosure has been described with reference to the embodiment shown in the drawings, this is merely exemplary, and those of ordinary skill in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present disclosure will have to be determined by the technical spirit of the appended claims.

Claims (20)

In the autonomous driving device,
a first sensor and a second sensor included in the lamp of the autonomous vehicle;
a processor electrically connected to the first sensor and the second sensor;
The processor, the autonomous vehicle,
Checking a free area in front of which the vehicle can move based on the first sensing data acquired through the first sensor,
Obtaining second sensing data for the free area based on the second sensor,
generating map information expressing the height of the road surface based on the second sensing data;
An autonomous driving device that controls to predict a road surface condition based on the pattern of the map information. The method of claim 1,
The first sensor includes a camera, and the second sensor includes a lidar. The method of claim 1,
The processor, the autonomous vehicle,
creating a plurality of grids including the free region;
Classifying the second sensing data to correspond to each of the generated grids,
Measuring the height of the road surface corresponding to each grid based on the classified second sensing data,
An autonomous driving device that controls to generate a grid map in which a height corresponding to each grid is expressed. 4. The method of claim 3,
at least one of a third sensor or a communication circuit for collecting driving information of the autonomous vehicle;
The processor, the autonomous vehicle,
In response to obtaining the driving information that satisfies the specified first condition, generating a grid having a first resolution,
In response to acquiring driving information that satisfies a specified second condition, the autonomous driving apparatus controls to generate a grid having a second resolution different from the first resolution. 5. The method of claim 4,
the communication circuit
An autonomous driving device that supports at least one of V2X or 5G communication methods. The method of claim 1,
at least one light source included in the lamp;
The processor, the autonomous vehicle,
In response to a road surface condition including an abnormal section being predicted, the autonomous driving device controls to adjust the light emission of the light source based on the abnormal section. 7. The method of claim 6,
The processor, the autonomous vehicle,
An autonomous driving apparatus controlling to adjust at least one of an irradiation direction of the light source and an irradiation intensity of the light source based on the abnormal section. 7. The method of claim 6,
The processor, the autonomous vehicle,
Obtaining third sensing data for the abnormal section by using the first sensor,
The autonomous driving apparatus controls to analyze the abnormal section based on the first sensed data and the third sensed data. 9. The method of claim 8,
The processor, the autonomous vehicle,
An autonomous driving device that controls to notify an analysis result of the abnormal section. The method of claim 1,
The processor, the autonomous vehicle,
mapping the second sensed data to the first sensed data,
The autonomous driving apparatus controls to generate the map information by acquiring the second sensing data corresponding to the free area among the mapped second sensing data. A method of operating an autonomous driving device, comprising:
checking a free area in front based on first sensed data through a first sensor included in a lamp of the autonomous vehicle;
acquiring second sensing data for the free area based on a second sensor included in the lamp;
generating map information representing the height of the road surface based on the second sensing data; and
and predicting a road surface condition based on the pattern of the map information. 12. The method of claim 11,
The first sensor comprises a camera and the second sensor comprises a lidar. 12. The method of claim 11,
The operation of generating the map information includes:
generating a plurality of gratings including the free region;
classifying the second sensing data to correspond to each of the generated grids;
measuring a height of a road surface corresponding to each of the grids based on the classified second sensing data; and
and generating a grid map in which a height corresponding to each grid is expressed. 14. The method of claim 13,
The operation of creating the grid is:
collecting driving information on the autonomous vehicle;
generating a grid having a first resolution in response to obtaining driving information that satisfies a specified first condition; and
and in response to obtaining driving information that satisfies a specified second condition, generating a grid having a second resolution different from the first resolution. 15. The method of claim 14,
The driving information is collected through at least one of a third sensor different from the first sensor and the second sensor, or a communication circuit of the autonomous driving device,
The communication circuit supports at least one of V2X, or 5G communication method. 12. The method of claim 11,
In response to a road surface condition including an abnormal section being predicted, controlling the light emission of the lamp based on the abnormal section. 17. The method of claim 16,
The operation of controlling the light emission of the lamp is,
and controlling at least one of an irradiation direction of the lamp and an irradiation intensity of the lamp based on the abnormal section. 17. The method of claim 16,
The operation of controlling the light emission of the lamp is,
obtaining third sensing data for the abnormal section by using the first sensor; and
and analyzing the abnormal section based on the first sensed data and the third sensed data. 19. The method of claim 18,
and notifying an analysis result for the abnormal section. 12. The method of claim 11,
The operation of acquiring the second sensed data includes:
mapping the second sensed data to the first sensed data; and
and acquiring the second sensed data corresponding to the free area from among the mapped second sensed data.

Images