KR102044703B1 - Autonomous vehicle and method of controlling the same - Google Patents

Abstract

The present invention relates to an autonomous vehicle and a control method thereof. An autonomous vehicle according to an embodiment of the present invention is a translation of the autonomous vehicle based on the at least two image frames and the camera to obtain at least two image frames for the external environment of the autonomous vehicle Motion and rotational motion.

Description

Autonomous vehicle and method of controlling the same}

The present invention relates to an autonomous vehicle and a control method thereof. More specifically, the present invention relates to a Simultaneous Localization And Mapping (SLAM) algorithm of an autonomous vehicle.

The vehicle is a device for moving in the direction desired by the user on board. An example is a car.

On the other hand, for the convenience of the user using the vehicle, various types of sensors, electronic devices, etc. are provided. In particular, research on the Advanced Driver Assistance System (ADAS) has been actively conducted for the user's driving convenience. Furthermore, development of autonomous vehicles is being actively performed.

On the other hand, the development of a technical field for generating a precise map of the external environment of the vehicle using the sensor provided in the vehicle is actively made. In particular, pose estimation of camera for lidar SLAM (Simultaneous Localization and Mapping) or visual SLAM is recognized as a core technology in generating a precise map. Camera pose estimation is to determine translation motion information and rotation motion information of a camera that is dynamically changing position.

In the autonomous vehicle, the visual SLAM system is a technology that processes frames acquired by a camera while the vehicle moves at 6-dof (6 degrees of freedom) and three-dimensional model the environment of the autonomous vehicle.

The conventional visual SLAM algorithm using a plurality of cameras uses a method of generating a plurality of models in each frame of a plurality of cameras, or integrating the frames obtained from the plurality of cameras and then generating a model. Accordingly, the visual SLAM algorithm according to the prior art has a problem in that a large amount of computation is required to be processed by the processor and the computation speed is slow.

Therefore, in the autonomous vehicle, there is a need for a visual SLAM algorithm that can reduce the amount of computation required by the processor and provide fast processing speed.

In order to solve the above problem, an embodiment of the present invention provides an autonomous vehicle that estimates optimal motion geometrically probabilistically by applying a model generated from an image having a relatively large number of feature points to another camera image in the same manner. There is a purpose.

The objects of the present invention are not limited to the above-mentioned objects, and other objects that are not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, an embodiment of the present invention comprises a camera for obtaining at least two image frames for the external environment of the autonomous vehicle; And a processor configured to calculate a translation motion and a rotation motion of the autonomous vehicle based on the obtained at least two image frames.

In an embodiment of the present invention, a plurality of cameras are provided, and the processor is configured to sample the image frames acquired by the plurality of cameras, and select a first camera based on the sampling result. do.

In an embodiment of the present disclosure, the processor may generate a random sample consensus (RANSAC) model from the image frame of the selected first camera.

Further, in an embodiment of the present invention, the processor applies the generated RANSAC model to an image frame of a second camera different from the selected first camera, and the cost of the RANSAC model applied to the image frame of the second camera. It is characterized by calculating.

Further, in an embodiment of the present invention, if it is determined that the calculated cost is greater than or equal to a predetermined value, the processor applies the RANSAC model to an image frame of a camera other than the first camera, and translates the autonomous vehicle. A translation motion and a rotation motion are calculated.

Further, in an embodiment of the present invention, the processor acquires, via the camera, a first key frame including a first image and a second key frame including a second image, and the first image and the second image. The 3D landmark is generated based on the feature points included in the feature.

Further, in an embodiment of the present invention, the processor further obtains a third key frame including a third image through the camera, and reprojection the generated 3D landmark to the third key frame. And calculating an error value between the projected value and the third image.

Further, in an embodiment of the present invention, the difference value includes a reprojection error term and a scale error term, and the processor corrects the reprojection error term and the size error term using a function optimization technique. Characterized in that.

In another embodiment, the autonomous vehicle further includes a wheel odometer, and the processor controls the wheel odometer to control the autonomous vehicle from the first key frame to the second key frame. The amount of movement is further detected.

Further, in the embodiment of the present invention, the processor is characterized in that for using the detected amount of movement to correct the magnitude error term.

Specific details of other embodiments are included in the detailed description and the drawings.

According to an embodiment of the present invention, there are one or more of the following effects.

First, according to an embodiment of the present invention, since the optimal motion is estimated by applying a model generated from an image having a relatively large number of feature points to another camera image in the same way, the processor of the autonomous vehicle must process it. This has the effect of reducing the amount of computation and providing fast processing speed.

Second, according to an embodiment of the present invention, since the actual amount of movement of the autonomous vehicle detected through the wheel odometer in the local bundle adjustment (LBA) is used for error correction, the translation of the autonomous vehicle is more accurately translated. There is an effect that can calculate the motion and rotation motion.

The effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a view showing the appearance of a vehicle according to an embodiment of the present invention.
2 is a view of the vehicle according to an embodiment of the present invention from various angles from the outside.
3 to 4 are views illustrating the interior of a vehicle according to an embodiment of the present invention.
5 to 6 are views referred to for describing an object according to an embodiment of the present invention.
7 is a block diagram referenced to describe a vehicle according to an embodiment of the present invention.
8 is a flowchart illustrating a SLAM algorithm of a vehicle according to an embodiment of the present invention.
9 is a flowchart illustrating a method of verifying a RANSAC model in a vehicle SLAM algorithm according to an embodiment of the present invention.
FIG. 10 is a diagram for describing a method in which a vehicle selects a camera using Probability Proportional to Size (PPS) sampling according to an exemplary embodiment of the present invention.
FIG. 11 illustrates a SLAM algorithm of a vehicle according to an embodiment of the present invention in pseudo-code.
12 is a flowchart illustrating a scale correcting Local Bundle Adjustment (scLBA) algorithm of a vehicle according to an embodiment of the present invention.
FIG. 13 is a conceptual diagram illustrating a scale correcting Local Bundle Adjustment (scLBA) algorithm of a vehicle according to an embodiment of the present invention. FIG.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, and the same or similar components are denoted by the same reference numerals regardless of the reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used in consideration of ease of specification, and do not have distinct meanings or roles from each other. In addition, in describing the embodiments disclosed herein, when it is determined that the detailed description of the related known technology may obscure the gist of the embodiments disclosed herein, the detailed description thereof will be omitted. In addition, the accompanying drawings are intended to facilitate understanding of the embodiments disclosed herein, but are not limited to the technical spirit disclosed herein by the accompanying drawings, all changes included in the spirit and scope of the present invention. It should be understood to include equivalents and substitutes.

Terms including ordinal numbers such as first and second may be used to describe various components, but the components are not limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprises" or "having" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

The vehicle described herein may be a concept including an automobile and a motorcycle. In the following, a vehicle is mainly described for a vehicle. The vehicle described herein may be a concept including both an internal combustion engine vehicle having an engine as a power source, a hybrid vehicle having an engine and an electric motor as a power source, an electric vehicle having an electric motor as a power source, and the like. In the following description, the left side of the vehicle means the left side of the driving direction of the vehicle, and the right side of the vehicle means the right side of the driving direction of the vehicle.

1 is a view showing the appearance of a vehicle according to an embodiment of the present invention.

2 is a view of the vehicle according to an embodiment of the present invention from various angles from the outside.

3 to 4 are views illustrating the interior of a vehicle according to an embodiment of the present invention.

5 to 6 are views referred to for describing an object according to an embodiment of the present invention.

7 is a block diagram referenced to describe a vehicle according to an embodiment of the present invention.

1 to 7, the vehicle 100 may include a wheel that rotates by a power source and a steering input device 510 for adjusting a traveling direction of the vehicle 100.

The vehicle 100 may be an autonomous vehicle. The vehicle 100 may be switched to an autonomous driving mode or a manual mode based on a user input. For example, the vehicle 100 may be switched from the manual mode to the autonomous driving mode or from the autonomous driving mode to the manual mode based on the received user input through the user interface device 200.

The vehicle 100 may be switched to the autonomous driving mode or the manual mode based on the driving situation information. The driving situation information may include at least one of object information, navigation information, and vehicle state information outside the vehicle.

For example, the vehicle 100 may be switched from the manual mode to the autonomous driving mode or from the autonomous driving mode to the manual mode based on the driving situation information generated by the object detecting apparatus 300. For example, the vehicle 100 may be switched from the manual mode to the autonomous driving mode or from the autonomous driving mode to the manual mode based on the driving situation information received through the communication device 400.

The vehicle 100 may switch from the manual mode to the autonomous driving mode or from the autonomous driving mode to the manual mode based on information, data, and signals provided from an external device.

When the vehicle 100 is driven in the autonomous driving mode, the autonomous vehicle 100 may be driven based on the driving system 700. For example, the autonomous vehicle 100 may be driven based on information, data, or signals generated by the driving system 710, the parking system 740, and the parking system 750.

When the vehicle 100 is driven in the manual mode, the autonomous vehicle 100 may receive a user input for driving through the driving manipulation apparatus 500. Based on a user input received through the driving manipulation apparatus 500, the vehicle 100 may be driven.

The overall length is the length from the front to the rear of the vehicle 100, the width is the width of the vehicle 100, and the height is the length from the bottom of the wheel to the roof. In the following description, the full length direction L is a direction in which the full length measurement of the vehicle 100 is a reference, the full width direction W is a direction in which the full width measurement of the vehicle 100 is a reference, and the total height direction H is a vehicle. It may mean the direction which is the reference of the height measurement of (100).

As illustrated in FIG. 7, the vehicle 100 includes a user interface device 200, an object detecting device 300, a communication device 400, a driving manipulation device 500, a vehicle driving device 600, and a traveling system. 700, a navigation system 770, a sensing unit 120, an interface unit 130, a memory 140, a control unit 170, and a power supply unit 190 may be included.

According to an exemplary embodiment, the vehicle 100 may further include other components in addition to the components described herein, or may not include some of the described components. The sensing unit 120 may include a state of the vehicle. Can sense. The sensing unit 120 may include an attitude sensor (for example, a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, and an inclination. Sensor, Weight Sensor, Heading Sensor, Gyro Sensor, Position Module, Vehicle Forward / Reverse Sensor, Battery Sensor, Fuel Sensor, Tire Sensor, Steering Sensor by Steering Wheel, Vehicle And an internal temperature sensor, an in-vehicle humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, and the like.

The sensing unit 120 includes vehicle attitude information, vehicle collision information, vehicle direction information, vehicle position information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward / reverse information, battery Acquire sensing signals for information, fuel information, tire information, vehicle lamp information, vehicle internal temperature information, vehicle internal humidity information, steering wheel rotation angle, vehicle external illumination, pressure applied to the accelerator pedal, pressure applied to the brake pedal, and the like. can do.

The sensing unit 120 may further include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), and a throttle position sensor. (TPS), TDC sensor, crank angle sensor (CAS), and the like.

The sensing unit 120 may generate vehicle state information based on the sensing data. The vehicle state information may be information generated based on data sensed by various sensors provided in the vehicle.

For example, the vehicle state information includes vehicle attitude information, vehicle speed information, vehicle tilt information, vehicle weight information, vehicle direction information, vehicle battery information, vehicle fuel information, vehicle tire pressure information, The vehicle may include steering information of the vehicle, vehicle indoor temperature information, vehicle indoor humidity information, pedal position information, vehicle engine temperature information, and the like.

The interface unit 130 may serve as a path to various types of external devices connected to the vehicle 100. For example, the interface unit 130 may include a port connectable with the mobile terminal, and may connect with the mobile terminal through the port. In this case, the interface unit 130 may exchange data with the mobile terminal.

Meanwhile, the interface unit 130 may serve as a path for supplying electrical energy to the connected mobile terminal. When the mobile terminal is electrically connected to the interface unit 130, under the control of the controller 170, the interface unit 130 may provide the mobile terminal with electrical energy supplied from the power supply unit 190.

The memory 140 is electrically connected to the controller 170. The memory 140 may store basic data for the unit, control data for controlling the operation of the unit, and input / output data. The memory 140 may be various storage devices such as a ROM, a RAM, an EPROM, a flash drive, a hard drive, and the like, in hardware. The memory 140 may store various data for overall operation of the vehicle 100, such as a program for processing or controlling the controller 170. According to an embodiment, the memory 140 may be integrally formed with the controller 170 or may be implemented as a subcomponent of the controller 170.

The controller 170 may control the overall operation of each unit in the vehicle 100. The controller 170 may be referred to as an electronic control unit (ECU). The power supply unit 190 may supply power required for the operation of each component under the control of the controller 170. In particular, the power supply unit 190 may receive power from a battery inside the vehicle.

One or more processors and controllers 170 included in vehicle 100 may include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable (FPGAs). Gate arrays, processors, controllers, micro-controllers, microprocessors, and other electrical units for performing other functions may be implemented.

In addition, the sensing unit 120, the interface unit 130, the memory 140, the power supply unit 190, the user interface device 200, the object detection device 300, the communication device 400, and the driving operation device 500. The vehicle driving apparatus 600, the driving system 700, and the navigation system 770 may have separate processors or may be integrated into the controller 170.

The user interface device 200 is a device for communicating with the vehicle 100 and a user. The user interface device 200 may receive a user input and provide the user with information generated in the vehicle 100. The vehicle 100 may implement user interfaces (UI) or user experience (UX) through the user interface device 200.

The user interface device 200 may include an input unit 210, an internal camera 220, a biometric detector 230, an output unit 250, and a processor 270. Each component of the user interface device 200 may be structurally and functionally separated from or integrated with the interface unit 130 described above.

According to an embodiment, the user interface device 200 may further include other components in addition to the described components, or may not include some of the described components.

The input unit 210 is for receiving information from a user, and the data collected by the input unit 210 may be analyzed by the processor 270 and processed as a user's control command.

The input unit 210 may be disposed in the vehicle. For example, the input unit 210 may include one area of a steering wheel, one area of an instrument panel, one area of a seat, one area of each pillar, and a door. one area of the door, one area of the center console, one area of the head lining, one area of the sun visor, one area of the windshield or of the window It may be disposed in one area or the like.

The input unit 210 may include a voice input unit 211, a gesture input unit 212, a touch input unit 213, and a mechanical input unit 214.

The voice input unit 211 may convert a user's voice input into an electrical signal. The converted electrical signal may be provided to the processor 270 or the controller 170. The voice input unit 211 may include one or more microphones.

The gesture input unit 212 may convert a user's gesture input into an electrical signal. The converted electrical signal may be provided to the processor 270 or the controller 170. The gesture input unit 212 may include at least one of an infrared sensor and an image sensor for detecting a user's gesture input.

According to an embodiment, the gesture input unit 212 may detect a 3D gesture input of the user. To this end, the gesture input unit 212 may include a light output unit or a plurality of image sensors for outputting a plurality of infrared light. The gesture input unit 212 may detect a user's 3D gesture input through a time of flight (TOF) method, a structured light method, or a disparity method.

The touch input unit 213 may convert a user's touch input into an electrical signal. The converted electrical signal may be provided to the processor 270 or the controller 170. The touch input unit 213 may include a touch sensor for detecting a user's touch input. According to an embodiment, the touch input unit 213 may be integrally formed with the display unit 251 to implement a touch screen. Such a touch screen may provide an input interface and an output interface between the vehicle 100 and the user.

The mechanical input unit 214 may include at least one of a button, a dome switch, a jog wheel, and a jog switch. The electrical signal generated by the mechanical input unit 214 may be provided to the processor 270 or the controller 170. The mechanical input unit 214 may be disposed on a steering wheel, a center fascia, a center console, a cockpit module, a door, or the like.

The processor 270 starts a learning mode of the vehicle 100 in response to user inputs to at least one of the voice input unit 211, the gesture input unit 212, the touch input unit 213, and the mechanical input unit 214 described above. can do. In the learning mode, the vehicle 100 may perform driving path learning and surrounding environment learning of the vehicle 100. The learning mode will be described in detail later with reference to the object detecting apparatus 300 and the driving system 700.

The internal camera 220 may acquire a vehicle interior image. The processor 270 may detect a state of the user based on the vehicle interior image. The processor 270 may acquire the gaze information of the user from the vehicle interior image. The processor 270 may detect a gesture of the user in the vehicle interior image.

The biometric detector 230 may acquire biometric information of the user. The biometric detector 230 may include a sensor for acquiring biometric information of the user, and may acquire fingerprint information, heartbeat information, etc. of the user using the sensor. Biometric information may be used for user authentication.

The output unit 250 is for generating output related to visual, auditory or tactile. The output unit 250 may include at least one of the display unit 251, the audio output unit 252, and the haptic output unit 253.

The display unit 251 may display graphic objects corresponding to various pieces of information. The display unit 251 is a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display (flexible) display, a 3D display, or an e-ink display.

The display unit 251 forms a layer structure or is integrally formed with the touch input unit 213 to implement a touch screen. The display unit 251 may be implemented as a head up display (HUD). When the display unit 251 is implemented as a HUD, the display unit 251 may include a projection module to output information through an image projected on a wind shield or a window. The display unit 251 may include a transparent display. The transparent display can be attached to the wind shield or window.

The transparent display may display a predetermined screen while having a predetermined transparency. In order to have transparency, transparent displays are transparent thin film electroluminescent (TFEL), transparent organic light-emitting diode (OLED), transparent liquid crystal (LCD), transparent transparent display, transparent LED (light emitting diode) display It may include at least one of. The transparency of the transparent display can be adjusted.

The user interface device 200 may include a plurality of display units 251a to 251g.

The display unit 251 may include one region of the steering wheel, one region 251a, 251b, and 251e of the instrument panel, one region 251d of the seat, one region 251f of each pillar, and one region of the door ( 251g), one area of the center console, one area of the head lining, one area of the sun visor, or may be implemented in one area 251c of the windshield and one area 251h of the window.

The sound output unit 252 converts an electrical signal provided from the processor 270 or the controller 170 into an audio signal and outputs the audio signal. To this end, the sound output unit 252 may include one or more speakers.

The haptic output unit 253 generates a tactile output. For example, the haptic output unit 253 may vibrate the steering wheel, the seat belt, and the seats 110FL, 110FR, 110RL, and 110RR so that the user may recognize the output.

The processor 270 may control the overall operation of each unit of the user interface device 200. According to an embodiment, the user interface device 200 may include a plurality of processors 270 or may not include the processor 270.

When the processor 270 is not included in the user interface device 200, the user interface device 200 may be operated under the control of the processor or the controller 170 of another device in the vehicle 100. The user interface device 200 may be referred to as a vehicle display device. The user interface device 200 may be operated under the control of the controller 170.

The object detecting apparatus 300 is a device for detecting an object located outside the vehicle 100. The object detecting apparatus 300 may generate object information based on the sensing data.

The object information may include information about the presence or absence of the object, location information of the object, distance information between the vehicle 100 and the object, and relative speed information between the vehicle 100 and the object. The object may be various objects related to the driving of the vehicle 100.

5 to 6, the object O includes a lane OB10, another vehicle OB11, a pedestrian OB12, a two-wheeled vehicle OB13, traffic signals OB14, OB15, light, a road, a structure, Speed bumps, features, animals and the like can be included.

The lane OB10 may be a driving lane, a lane next to the driving lane, and a lane in which an opposite vehicle travels. The lane OB10 may be a concept including left and right lines forming a lane.

The other vehicle OB11 may be a vehicle that is driving around the vehicle 100. The other vehicle may be a vehicle located within a predetermined distance from the vehicle 100. For example, the other vehicle OB11 may be a vehicle that precedes or follows the vehicle 100.

The pedestrian OB12 may be a person located near the vehicle 100. The pedestrian OB12 may be a person located within a predetermined distance from the vehicle 100. For example, the pedestrian OB12 may be a person located on a sidewalk or a roadway.

The two-wheeled vehicle OB13 may be a vehicle that is positioned around the vehicle 100 and moves using two wheels. The motorcycle OB13 may be a vehicle having two wheels located within a predetermined distance from the vehicle 100. For example, the motorcycle OB13 may be a motorcycle or a bicycle located on sidewalks or roadways.

The traffic signal may include a traffic light OB15, a traffic sign OB14, and a pattern or text drawn on a road surface. The light may be light generated by a lamp provided in another vehicle. The light, can be light generated from the street light. The light may be sunlight. The road may include a road surface, a curve, an uphill slope, a slope downhill, or the like. The structure may be an object located around a road and fixed to the ground. For example, the structure may include a street lamp, a roadside tree, a building, a power pole, a traffic light, a bridge. The features may include mountains, hills, and the like.

On the other hand, the object may be classified into a moving object and a fixed object. For example, the moving object may be a concept including another vehicle and a pedestrian. For example, the fixed object may be a concept including a traffic signal, a road, and a structure.

The object detecting apparatus 300 may include a camera 310, a radar 320, a lidar 330, an ultrasonic sensor 340, an infrared sensor 350, and a processor 370. Each component of the object detecting apparatus 300 may be structurally and functionally separated or integrated with the sensing unit 120 described above.

According to an embodiment, the object detecting apparatus 300 may further include other components in addition to the described components, or may not include some of the described components.

The camera 310 may be located at a suitable place outside the vehicle to acquire an image outside the vehicle. The camera 310 may be a mono camera, a stereo camera 310a, an around view monitoring (AVM) camera 310b, or a 360 degree camera.

The camera 310 may acquire location information of the object, distance information with respect to the object, or relative speed information with the object by using various image processing algorithms.

For example, the camera 310 may obtain distance information and relative speed information with respect to the object based on the change in the object size over time in the acquired image.

For example, the camera 310 may acquire distance information and relative velocity information with respect to an object through a pin hole model, road surface profiling, or the like.

For example, the camera 310 may obtain distance information and relative speed information with respect to the object based on the disparity information in the stereo image acquired by the stereo camera 310a.

For example, the camera 310 may be disposed in close proximity to the front windshield in the interior of the vehicle in order to acquire an image in front of the vehicle. Alternatively, the camera 310 may be disposed around the front bumper or the radiator grille.

For example, the camera 310 may be disposed in close proximity to the rear glass in the interior of the vehicle to acquire an image of the rear of the vehicle. Alternatively, the camera 310 may be disposed around the rear bumper, the trunk, or the tail gate.

For example, the camera 310 may be disposed in close proximity to at least one of the side windows in the interior of the vehicle to acquire an image of the vehicle side. Alternatively, the camera 310 may be arranged around the side mirror, fender or door.

The camera 310 may provide the obtained image to the processor 370.

The radar 320 may include an electromagnetic wave transmitter and a receiver. The radar 320 may be implemented in a pulse radar method or a continuous wave radar method in terms of radio wave firing principle. The radar 320 may be implemented by a frequency modulated continuous wave (FSCW) method or a frequency shift keying (FSK) method according to a signal waveform among continuous wave radar methods.

The radar 320 detects an object based on a time of flight (TOF) method or a phase-shift method based on electromagnetic waves, and detects the position of the detected object, distance to the detected object, and relative velocity. Can be detected.

The radar 320 may be disposed at an appropriate position outside the vehicle to detect an object located in front, rear, or side of the vehicle.

The lidar 330 may include a laser transmitter and a receiver. The lidar 330 may be implemented in a time of flight (TOF) method or a phase-shift method.

The lidar 330 may be implemented as driven or non-driven. When implemented in a driving manner, the lidar 330 may be rotated by a motor and detect an object around the vehicle 100. When implemented in a non-driven manner, the lidar 330 may detect an object located within a predetermined range with respect to the vehicle 100 by optical steering. The vehicle 100 may include a plurality of non-driven lidars 330.

The lidar 330 detects an object based on a time of flight (TOF) method or a phase-shift method using laser light, and detects an object, a position of the detected object, a distance from the detected object, and Relative speed can be detected. The lidar 330 may be disposed at an appropriate position outside the vehicle to detect an object located in front, rear, or side of the vehicle.

The ultrasonic sensor 340 may include an ultrasonic transmitter and a receiver. The ultrasonic sensor 340 may detect an object based on the ultrasonic wave, and detect a position of the detected object, a distance to the detected object, and a relative speed. The ultrasonic sensor 340 may be disposed at an appropriate position outside the vehicle to detect an object located in front, rear, or side of the vehicle.

The infrared sensor 350 may include an infrared transmitter and a receiver. The infrared sensor 340 may detect an object based on infrared light, and detect a position of the detected object, a distance to the detected object, and a relative speed. The infrared sensor 350 may be disposed at an appropriate position outside the vehicle to detect an object located in front, rear, or side of the vehicle.

The processor 370 may control overall operations of each unit of the object detecting apparatus 300. The processor 370 compares the data sensed by the camera 310, the radar 320, the lidar 330, the ultrasonic sensor 340, and the infrared sensor 350 with previously stored data to detect or classify an object. can do.

The processor 370 may detect and track the object based on the obtained image. The processor 370 may perform operations such as calculating a distance to an object and calculating a relative speed with the object through an image processing algorithm.

For example, the processor 370 may acquire distance information and relative speed information with respect to the object based on the change in the object size over time in the obtained image.

For example, the processor 370 may acquire distance information and relative velocity information with respect to an object through a pin hole model, road surface profiling, or the like.

For example, the processor 370 may obtain distance information and relative speed information with the object based on the disparity information in the stereo image acquired by the stereo camera 310a.

The processor 370 may detect and track the object based on the reflected electromagnetic wave reflected by the transmitted electromagnetic wave to the object. The processor 370 may perform an operation such as calculating a distance from the object, calculating a relative speed with the object, and the like based on the electromagnetic waves.

The processor 370 may detect and track the object based on the reflected laser light reflected by the transmitted laser back to the object. The processor 370 may perform an operation such as calculating a distance from the object, calculating a relative speed with the object, and the like based on the laser light.

The processor 370 may detect and track the object based on the reflected ultrasound, in which the transmitted ultrasound is reflected by the object and returned. The processor 370 may perform an operation such as calculating a distance from the object, calculating a relative speed with the object, and the like based on the ultrasound.

The processor 370 may detect and track the object based on the reflected infrared light from which the transmitted infrared light is reflected back to the object. The processor 370 may perform an operation such as calculating a distance to the object, calculating a relative speed with the object, and the like based on the infrared light.

As described above, when the learning mode of the vehicle 100 is initiated in response to a user input to the input unit 210, the processor 370 may include a camera 310, a radar 320, a lidar 330, and an ultrasonic sensor. The data sensed by the 340 and the infrared sensor 350 may be stored in the memory 140.

Each step of the learning mode based on the analysis of the stored data and an operation mode following the learning mode will be described in detail with reference to the operation system 700. According to an embodiment, the object detecting apparatus 300 In addition, the processor 370 may or may not include the processor 370. For example, each of the camera 310, the radar 320, the lidar 330, the ultrasonic sensor 340, and the infrared sensor 350 may individually include a processor.

When the processor 370 is not included in the object detecting apparatus 300, the object detecting apparatus 300 may be operated under the control of the processor or the controller 170 of the apparatus in the vehicle 100. The object detecting apparatus 300 may be operated under the control of the controller 170.

The communication device 400 is a device for performing communication with an external device. Here, the external device may be another vehicle, a mobile terminal or a server. The communication device 400 may include at least one of a transmit antenna, a receive antenna, a radio frequency (RF) circuit capable of implementing various communication protocols, and an RF element to perform communication.

The communication device 400 includes a short range communication unit 410, a location information unit 420, a V2X communication unit 430, an optical communication unit 440, a broadcast transmission / reception unit 450, an ITS (Intelligent Transport Systems) communication unit 460, and a processor. 470 may include. According to an embodiment, the communication device 400 may further include other components in addition to the described components, or may not include some of the described components.

The short range communication unit 410 is a unit for short range communication. The short range communication unit 410 may include Bluetooth®, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Near Field Communication (NFC), and Wi-Fi (Wireless). Local area communication may be supported using at least one of Fidelity, Wi-Fi Direct, and Wireless Universal Serial Bus (USB) technologies. The short range communication unit 410 may form short range wireless networks to perform short range communication between the vehicle 100 and at least one external device.

The location information unit 420 is a unit for obtaining location information of the vehicle 100. For example, the location information unit 420 may include a global positioning system (GPS) module or a differential global positioning system (DGPS) module.

The V2X communication unit 430 is a unit for performing wireless communication with a server (V2I: Vehicle to Infra), another vehicle (V2V: Vehicle to Vehicle), or a pedestrian (V2P: Vehicle to Pedestrian). The V2X communication unit 430 may include an RF circuit that can implement a communication with the infrastructure (V2I), an inter-vehicle communication (V2V), and a communication with the pedestrian (V2P).

The optical communication unit 440 is a unit for performing communication with an external device via light. The optical communication unit 440 may include an optical transmitter that converts an electrical signal into an optical signal and transmits the external signal to the outside, and an optical receiver that converts the received optical signal into an electrical signal. According to an embodiment, the light emitting unit may be formed to be integrated with the lamp included in the vehicle 100.

The broadcast transceiver 450 is a unit for receiving a broadcast signal from an external broadcast management server or transmitting a broadcast signal to a broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, and a data broadcast signal.

The ITS communication unit 460 may exchange information, data, or signals with the traffic system. The ITS communication unit 460 may provide the obtained information and data to the transportation system. The ITS communication unit 460 may receive information, data, or a signal from a traffic system. For example, the ITS communication unit 460 may receive road traffic information from the traffic system and provide the road traffic information to the control unit 170. For example, the ITS communication unit 460 may receive a control signal from a traffic system and provide the control signal to a processor provided in the controller 170 or the vehicle 100.

The processor 470 may control the overall operation of each unit of the communication device 400. According to an embodiment, the communication device 400 may include a plurality of processors 470 or may not include the processor 470. When the processor 470 is not included in the communication device 400, the communication device 400 may be operated under the control of the processor or the controller 170 of another device in the vehicle 100.

Meanwhile, the communication device 400 may implement a vehicle display device together with the user interface device 200. In this case, the vehicle display device may be called a telematics device or an audio video navigation (AVN) device. The communication device 400 may be operated under the control of the controller 170.

The driving operation apparatus 500 is a device that receives a user input for driving. In the manual mode, the vehicle 100 may be driven based on a signal provided by the driving manipulation apparatus 500. The driving manipulation apparatus 500 may include a steering input apparatus 510, an acceleration input apparatus 530, and a brake input apparatus 570.

The steering input device 510 may receive a driving direction input of the vehicle 100 from the user. The steering input device 510 is preferably formed in a wheel shape to enable steering input by rotation. According to an embodiment, the steering input device may be formed in the form of a touch screen, a touch pad, or a button.

The acceleration input device 530 may receive an input for accelerating the vehicle 100 from a user. The brake input device 570 may receive an input for deceleration of the vehicle 100 from a user. The acceleration input device 530 and the brake input device 570 are preferably formed in the form of a pedal. According to an embodiment, the acceleration input device or the brake input device may be formed in the form of a touch screen, a touch pad, or a button.

The driving manipulation apparatus 500 may be operated under the control of the controller 170.

The vehicle drive device 600 is a device that electrically controls the driving of various devices in the vehicle 100. The vehicle driving apparatus 600 may include a power train driver 610, a chassis driver 620, a door / window driver 630, a safety device driver 640, a lamp driver 650, and an air conditioning driver 660. Can be. According to an embodiment, the vehicle driving apparatus 600 may further include other components in addition to the described components, or may not include some of the described components. On the other hand, the vehicle driving device 600 may include a processor. Each unit of the vehicle drive apparatus 600 may each include a processor individually.

The power train driver 610 may control the operation of the power train device. The power train driver 610 may include a power source driver 611 and a transmission driver 612.

The power source driver 611 may control the power source of the vehicle 100. For example, when the fossil fuel-based engine is a power source, the power source driver 610 may perform electronic control of the engine. Thereby, the output torque of an engine, etc. can be controlled. The power source drive unit 611 can adjust the engine output torque under the control of the control unit 170.

For example, when the electric energy based motor is a power source, the power source driver 610 may control the motor. The power source driver 610 may adjust the rotational speed, torque, and the like of the motor under the control of the controller 170.

The transmission driver 612 may control the transmission. The transmission driver 612 can adjust the state of the transmission. The transmission drive part 612 can adjust the state of a transmission to forward D, backward R, neutral N, or parking P. FIG. On the other hand, when the engine is a power source, the transmission drive unit 612 can adjust the bite state of the gear in the forward D state.

The chassis driver 620 may control the operation of the chassis device. The chassis driver 620 may include a steering driver 621, a brake driver 622, and a suspension driver 623.

The steering driver 621 may perform electronic control of a steering apparatus in the vehicle 100. The steering driver 621 may change the traveling direction of the vehicle.

The brake driver 622 may perform electronic control of a brake apparatus in the vehicle 100. For example, the speed of the vehicle 100 may be reduced by controlling the operation of the brake disposed on the wheel.

On the other hand, the brake drive unit 622 can individually control each of the plurality of brakes. The brake driver 622 may control the braking force applied to the plurality of wheels differently.

The suspension driver 623 may perform electronic control of a suspension apparatus in the vehicle 100. For example, when there is a curvature on the road surface, the suspension driver 623 may control the suspension device to control the vibration of the vehicle 100 to be reduced. Meanwhile, the suspension driver 623 may individually control each of the plurality of suspensions.

The door / window driver 630 may perform electronic control of a door apparatus or a window apparatus in the vehicle 100. The door / window driver 630 may include a door driver 631 and a window driver 632.

The door driver 631 may control the door apparatus. The door driver 631 may control opening and closing of the plurality of doors included in the vehicle 100. The door driver 631 may control the opening or closing of a trunk or a tail gate. The door driver 631 may control the opening or closing of the sunroof.

The window driver 632 may perform electronic control of the window apparatus. The opening or closing of the plurality of windows included in the vehicle 100 may be controlled.

The safety device driver 640 may perform electronic control of various safety apparatuses in the vehicle 100. The safety device driver 640 may include an airbag driver 641, a seat belt driver 642, and a pedestrian protection device driver 643.

The airbag driver 641 may perform electronic control of an airbag apparatus in the vehicle 100. For example, the airbag driver 641 may control the airbag to be deployed when the danger is detected.

The seat belt driver 642 may perform electronic control of a seatbelt apparatus in the vehicle 100. For example, the seat belt driver 642 may control the passenger to be fixed to the seats 110FL, 110FR, 110RL, and 110RR by using the seat belt when detecting a danger.

The pedestrian protection device driver 643 may perform electronic control of the hood lift and the pedestrian airbag. For example, the pedestrian protection device driver 643 may control the hood lift up and the pedestrian air bag to be deployed when the collision with the pedestrian is detected.

The lamp driver 650 may perform electronic control of various lamp apparatuses in the vehicle 100.

The air conditioning driver 660 may perform electronic control of an air conditioner in the vehicle 100. For example, when the temperature inside the vehicle is high, the air conditioning driver 660 may control the air conditioning apparatus to operate to supply cool air to the inside of the vehicle.

The vehicle driving apparatus 600 may include a processor. Each unit of the vehicle drive apparatus 600 may each include a processor individually. The vehicle driving apparatus 600 may be operated under the control of the controller 170.

The travel system 700 is a system for controlling various travels of the vehicle 100. The navigation system 700 can be operated in an autonomous driving mode.

The travel system 700 can include a travel system 710, a parking system 740, and a parking system 750. In some embodiments, the navigation system 700 may further include other components in addition to the described components, or may not include some of the described components. Meanwhile, the driving system 700 may include a processor. Each unit of the navigation system 700 may each include a processor individually.

On the other hand, the driving system 700 may control the driving of the autonomous driving mode based on the learning. In this case, the learning mode and the operation mode on the premise that the learning is completed may be performed. A method of the processor of the driving system 700 to perform a learning mode and an operating mode will be described below.

The learning mode may be performed in the manual mode described above. In the learning mode, the processor of the driving system 700 may perform driving path learning and surrounding environment learning of the vehicle 100.

The driving route learning may include generating map data on a route on which the vehicle 100 travels. In particular, the processor of the driving system 700 may generate map data based on information detected by the object detecting apparatus 300 while the vehicle 100 travels from the starting point to the destination.

The surrounding environment learning may include storing and analyzing information about the surrounding environment of the vehicle 100 in the driving process and the parking process of the vehicle 100. In particular, the processor of the driving system 700 may detect information detected by the object detecting apparatus 300 during the parking process of the vehicle 100, for example, location information of the parking space, size information, fixed (or not fixed). Information about the surrounding environment of the vehicle 100 may be stored and analyzed based on information such as obstacle information.

The operation mode may be performed in the autonomous driving mode described above. The operation mode will be described on the premise that the driving route learning or the surrounding environment learning is completed through the learning mode.

The operation mode may be performed in response to a user input through the input unit 210, or may be automatically performed when the vehicle 100 reaches a driving path and a parking space where learning is completed.

The operating mode is a semi autonomous operating mode that requires some user's manipulation of the drive manipulator 500 and a full-autonomous operation requiring no user's manipulation of the drive manipulator 500. May include a fully autonomous operating mode.

Meanwhile, according to an exemplary embodiment, the processor of the driving system 700 may control the driving system 710 in the operation mode to drive the vehicle 100 along the driving path where learning is completed.

Meanwhile, according to an exemplary embodiment, the processor of the driving system 700 may control the parking system 740 in the operation mode to release the parked vehicle 100 from the parking space where the learning is completed.

Meanwhile, according to an exemplary embodiment, the processor of the driving system 700 may control the parking system 750 in the operation mode to park the vehicle 100 from the current position to the parking space where the learning is completed. When the driving system 700 is implemented in software, the driving system 700 may be a lower concept of the controller 170.

In some embodiments, the driving system 700 may include a user interface device 270, an object detecting device 300, a communication device 400, a driving manipulation device 500, a vehicle driving device 600, and a navigation system. 770, the sensing unit 120, and the control unit 170 may include a concept including at least one.

The traveling system 710 may perform driving of the vehicle 100. The driving system 710 may receive navigation information from the navigation system 770, provide a control signal to the vehicle driving apparatus 600, and perform driving of the vehicle 100.

The driving system 710 may receive object information from the object detecting apparatus 300 and provide a control signal to the vehicle driving apparatus 600 to perform driving of the vehicle 100. The driving system 710 may receive a signal from an external device through the communication device 400, provide a control signal to the vehicle driving device 600, and perform driving of the vehicle 100.

The driving system 710 may include a user interface device 270, an object detection device 300, a communication device 400, a driving manipulation device 500, a vehicle driving device 600, a navigation system 770, and a sensing unit ( At least one of the 120 and the controller 170 may be a system concept for driving the vehicle 100. The driving system 710 may be referred to as a vehicle driving control device.

The taking-out system 740 may perform taking out of the vehicle 100. The taking-out system 740 may receive navigation information from the navigation system 770, provide a control signal to the vehicle driving apparatus 600, and perform take-out of the vehicle 100.

The taking-out system 740 may receive the object information from the object detecting apparatus 300, provide a control signal to the vehicle driving apparatus 600, and perform take-out of the vehicle 100.

The taking-off system 740 may receive a signal from an external device through the communication device 400, provide a control signal to the vehicle driving apparatus 600, and perform take-out of the vehicle 100.

The car leaving system 740 includes a user interface device 270, an object detecting device 300 and a communication device 400, a driving control device 500, a vehicle driving device 600, a navigation system 770, and a sensing unit ( Including at least one of the controller 120 and the controller 170, the concept of a system that performs the taking out of the vehicle 100 may be performed.

Such a car leaving system 740 may be referred to as a vehicle parking control device.

The parking system 750 may perform parking of the vehicle 100. The parking system 750 may receive navigation information from the navigation system 770, provide a control signal to the vehicle driving apparatus 600, and perform parking of the vehicle 100.

The parking system 750 may receive the object information from the object detecting apparatus 300, provide a control signal to the vehicle driving apparatus 600, and perform parking of the vehicle 100.

The parking system 750 may receive a signal from an external device through the communication device 400, provide a control signal to the vehicle driving device 600, and perform parking of the vehicle 100.

The parking system 750 includes a user interface device 270, an object detection device 300 and a communication device 400, a driving operation device 500, a vehicle driving device 600, a navigation system 770, and a sensing unit ( At least one of the 120 and the controller 170 may be a system concept for parking the vehicle 100.

Such a parking system 750 may be referred to as a vehicle parking control device.

The navigation system 770 can provide navigation information. The navigation information may include at least one of map information, set destination information, route information according to the destination setting, information on various objects on the route, lane information, and current location information of the vehicle.

The navigation system 770 may include a memory and a processor. The memory may store navigation information. The processor may control the operation of the navigation system 770.

According to an embodiment, the navigation system 770 may receive information from an external device through the communication device 400 and update the pre-stored information. According to an embodiment, the navigation system 770 may be classified as a subcomponent of the user interface device 200.

For Visual SLAM RANSAC  algorithm

8 to 11 are diagrams for explaining the SLAM algorithm according to an embodiment of the present invention. Prior to the description of FIG. 8, first, Simultaneous Localization and Mapping (SLAM) and Random sample consensus (RANSAC) will be described. Hereinafter, the processor 170 of the autonomous vehicle may be understood as the same component as the controller 170 illustrated in FIG. 7.

SLAM detects a motion vector in image frames and estimates a posture of a moving object. The SLAM refers to a technology for generating an accurate map of an object's external environment. In general, lidar SLAM and visual SLAM are well known. The lidar SLAM uses a method of detecting a point cloud through a time of flight (TOF) sensor. On the other hand, visual SLAM uses the intensity of an image included in an image frame. lidar SLAM and visual SLAM are suitable techniques to be performed in a space where a corner point or a feature point can be sufficiently detected, for example, an indoor space or a city center.

RANSAC is an iterative method of estimating the parameters of a mathematical model from a set of observation data including outliers. RANSAC can be understood as a mathematical method used to implement the SLAM technique described above.

On the other hand, the visual SLAM system can track the position and orientation of the camera with respect to the 3D model. Keyframe-based visual SLAM systems can process frames selected separately from the input image stream or supply. Keyframe-based visual SLAM systems generate 3D feature maps from normal camera motion.

On the other hand, the moving object in the three-dimensional space can be modeled in 3-rotation and 3-translation (6 degrees of freedom, 6-dof). 3-rotation may be defined as a rotational motion including yaw, pitch, and roll, and 3-translation may be defined as translational motion on the x-axis, y-axis, and z-axis.

8 to 11 proposes a method of implementing visual SLAM using a plurality of image frames from a plurality of cameras (for example, four) provided in an autonomous vehicle. In particular, the camera of the autonomous vehicle according to the embodiment of the present invention obtains at least two image frames for the external environment, and the processor is configured to perform the translational motion and rotational motion of the autonomous vehicle based on the obtained at least two image frames. Calculate.

On the other hand, the camera may be located at a suitable place outside the autonomous vehicle in order to obtain an external image of the autonomous vehicle. The camera may be a mono camera, a stereo camera, an around view monitoring (AVM) camera, or a 360 degree camera.

The image frame obtained through the camera includes a feature point such as an interface of the object and a corner point. There are various conventional algorithms for detecting the aforementioned feature points.

The SLAM algorithm according to the prior art first uses a method of comparing a plurality of frames to detect feature points with little change, and extracting about 200-300 matching feature points. In addition, the SLAM algorithm according to the related art using a plurality of cameras uses a method of generating a plurality of models in a frame of each of the plurality of cameras or integrating the frames from the plurality of cameras and then generating a model.

On the other hand, the SLAM algorithm according to an embodiment of the present invention provides a method for estimating optimal motion in geometrical probability by applying a model generated from an image having a large number of feature points to another camera image in the same manner.

That is, since there is an error due to the position of the camera or other factors, the processor 170 of the autonomous vehicle according to the embodiment of the present invention generates a base model from the frame of the camera where the feature points are most likely to be detected. Then, the generated base model is verified using the frames of the remaining cameras.

Referring to FIG. 8 to be described in more detail the SLAM algorithm according to an embodiment of the present invention. In order to describe each step of FIG. 8 in more detail, FIGS. 9 to 11 will be described together.

According to step 810 of FIG. 8, the processor 170 of the autonomous vehicle selects the first camera through sampling of at least two image frames obtained from at least one camera. The sampling may be, for example, Probability Proportional to Size (PPS) sampling. Hereinafter, PPS sampling will be described in detail with reference to FIG. 10.

Referring to FIG. 10, the processor 170 of an autonomous vehicle according to an exemplary embodiment of the present invention describes a method of selecting a first camera using Probability Proportional to Size (PPS) sampling.

First, it is assumed that autonomous vehicles have A to D cameras at different positions. In FIG. 10, area A means sample size of camera A, area B means sample size of camera B, area C means sample size of camera C, and area D means sample size of camera D. .

The processor 170 of the autonomous vehicle samples the variable x uniformly in a sample space from 0 to 1 using PPS sampling. PPS sampling is a method of sampling x with a greater probability in a cluster having a large sample size. Referring to FIG. 10, it can be seen that x has a high probability of sampling in the D region. Next, the processor 170 of the autonomous vehicle selects a cluster including x. The selected cluster means a camera corresponding thereto. Referring to FIG. 10, it can be seen that the D camera is selected.

Meanwhile, with reference to FIG. 10, the processor 170 of the autonomous vehicle according to another embodiment of the present invention may select the first camera based on the motion attribute of the autonomous vehicle.

If the motion attribute of the autonomous vehicle is translation motion, a frame having more feature points is obtained through the camera mounted on the side rather than the camera mounted on the front / rear. That is, since the camera mounted on the side is advantageous for the translational motion, the processor of the autonomous vehicle selects the camera mounted on the side as the first camera if the motion attribute of the autonomous vehicle is the translational motion.

On the other hand, as in the corner section, if the motion attribute of the autonomous vehicle is rotation (rotation) motion, it is difficult to distinguish whether the motion of the autonomous vehicle is a translational motion or a rotational motion through the camera mounted on the side. That is, since the camera mounted on the front / rear is advantageous for the rotational motion, the processor of the autonomous vehicle selects the camera mounted on the front / rear as the first camera when the motion attribute of the autonomous vehicle is the rotational motion.

On the other hand, the motion properties of autonomous vehicles generally include both translational and rotational motions. Accordingly, the processor 170 of the autonomous vehicle according to another embodiment of the present invention applies a large weight to the camera corresponding to the motion attribute that occupies a higher ratio in the motion attribute of the autonomous vehicle to the first camera. You can also choose. Furthermore, the processor of the autonomous vehicle according to another embodiment of the present invention may determine the motion attribute of the autonomous vehicle through the wheel odometer.

Next, according to step 820 of FIG. 8, the processor 170 of the autonomous vehicle samples at least one image vector and a landmark pair. In operation 830 of FIG. 8, the processor 170 of the autonomous vehicle generates a RANSAC model calculated by the RANSAC algorithm. For example, the RANSAC algorithm may be a P3P-RANSAC algorithm.

Referring to step 830 of FIG. 8, the processor 170 of the autonomous vehicle according to the embodiment of the present invention performs a process of searching for a parameter most suitable for the R (θ) model in the frame. In doing so, the processor 170 uses at least two image frames. For example, the processor 170 generates a RANSAC model from the first image frame acquired through the first camera, and then translates and translates the translation motion and rotation based on the second image frame acquired through the first camera. rotation) Calculate the motion.

How the processor 170 generates a RANSAC model from a frame may be represented by Equation 1 below. In Equation 1, x and y are coordinates of a feature point in the first image frame, x 'and y' are coordinates of a feature point in the second image frame, R represents rotation motion, and T is Represents a translation motion.

Next, according to step 840 of FIG. 8, the processor 170 of the autonomous vehicle verifies the RANSAC model generated in step 830. Step 840 of FIG. 8 will be described later in more detail with reference to FIG. 9.

9 is a flowchart illustrating a method of verifying a RANSAC model in a vehicle SLAM algorithm according to an embodiment of the present invention. First, according to step 910 of FIG. 9, the processor 170 of the autonomous vehicle acquires an image vector and a landmark pair from a camera.

According to step 920 of FIG. 9, the processor 170 of the autonomous vehicle applies the RANSAC model generated in step 830 of FIG. 8 to a frame of another camera. The plurality of cameras mounted on the autonomous vehicle are different in their respective positions. Accordingly, in operation 920, the processor 170 performs an operation of aligning a plurality of cameras with motion of the same condition.

According to step 930 of FIG. 9, the processor 170 of the autonomous vehicle calculates a cost of a RANSAC model applied to a frame of another camera.

In operation 940 of FIG. 9, the processor 170 of the autonomous vehicle determines whether the calculated cost satisfies a predetermined value or more. If the cost satisfies the predetermined value or more, the processor 170 sets the RANSAC model generated in step 830 of FIG. 8 as an optimal model. The processor 170 of the self-driving vehicle according to the embodiment of the present invention is designed to iterate the loop illustrated in FIG. 9 until a cost that satisfies a predetermined value or more is calculated.

Meanwhile, in step 940 of FIG. 9, the processor 170 determines whether the cost satisfies a predetermined value or more, and terminates the loop accordingly, but according to another embodiment of the present invention, the processor 170 You can also set other conditions for terminating an iteration loop. For example, the repetition loop may be terminated when the number of inliers of the RANSAC model applied to the frame of another camera is equal to or less than a predetermined number or satisfies the maximum number of repetitions of the repetition loop.

FIG. 11 illustrates a SLAM algorithm of a vehicle according to an embodiment of the present invention in pseudo-code.

Part (a) of FIG. 11 corresponds to steps 810 to 830 of FIG. 8. Part (b) of FIG. 11 corresponds to step 840 of FIG. 8 and steps 910 to 930 of FIG. 9. Part (c) of FIG. 11 corresponds to step 940 of FIG. The code shown in FIG. 11 is exemplary, and the scope of rights of the SLAM algorithm according to the embodiment of the present invention is not limited thereto.

The SLAM algorithm of the present invention, which can be implemented with the code shown in FIG. 11, is a technique capable of estimating optimal motion in geometric probabilities by applying the same model generated in a frame having a relatively large number of feature points to another camera frame. It works.

The embodiments of the present invention described with reference to FIGS. 8 to 11 relate to motion detection based on frames when a plurality of cameras is provided, that is, when there is a dependency on the number of cameras. 12 to 13, a method of performing motion detection in a monocular camera regardless of the number of cameras, that is, when there is no dependency on the number of cameras according to an embodiment of the present invention will be described.

scale correcting Local Bundle Adjustment scLBA )

12 to 13 are diagrams for explaining the SLAM algorithm of an autonomous vehicle according to an embodiment of the present invention. More specifically, FIG. 12 is a flowchart illustrating a scale correcting Local Bundle Adjustment (scLBA) algorithm, and FIG. 13 is a conceptual diagram illustrating the flowchart of FIG. 12. First, prior to the description of FIG. 12, a description will be given of LBA (Local Bundle Adjustment).

Local Bundle Adjustment (LBA) is a method of nonlinear optimization of information and estimated posture of three-dimensional points on an environment map that is close to the current posture in time and space. In general, in the motion estimation process, the posture includes a geometric error, and the error accumulates over time, thereby degrading the accuracy of the posture measurement. LBA is used to solve this problem.

The LBA algorithm according to the prior art uses a method of generating a disparity map in consideration of the installation position of the stereo camera in order to 3D model the surrounding environment through the monocular camera. In other words, the LBA algorithm according to the related art may be referred to as a method of using parallax between successive frames.

However, the LBA algorithm according to the prior art has a problem that drift occurs when the moving path becomes long because the optimization based on the estimated value is performed. In addition, since the LBA algorithm according to the prior art does not use the actual size information of the route, there is a fundamental limitation that there is no unit in the estimated route and the environment map.

Hereinafter, a scLBA (scale correcting local bundle adjustment) algorithm performed by a processor of an autonomous vehicle according to an embodiment of the present invention will be described with reference to FIGS. 12 to 13.

Referring to FIG. 12, the processor 170 of the self-driving vehicle to which the scLBA algorithm is applied according to an embodiment of the present invention uses at least three key frames through a camera mounted on the autonomous vehicle before step 1210 of FIG. 12 is performed. key frame).

Referring to FIG. 13, the detected frames are a first key frame 1301, a second key frame 1302, and a third key frame 1303. The first key frame 1301 includes at least one first image 1310, the second key frame 1302 includes at least one second image 1320, and the third key frame 1303 is At least one third image 1330 is included.

Referring to steps 1210 and 13 of FIG. 12, the processor 170 of the autonomous vehicle generates the 3D landmark 1350 using the first key frame 1301 and the second key frame 1302.

In FIG. 13, the first image 1310 and the second image 1320 include at least one feature point. The processor 170 of the autonomous vehicle extracts the feature points 1340 corresponding to each other from the first image 1310 and the second image 1320, and generates the 3D landmark 1350 based on the extracted feature points 1340. Create

Next, referring to steps 1220 and 13 of FIG. 12, the processor 170 of the autonomous vehicle reprojects the 3D landmark 1350 generated according to step 1210 of FIG. 12 to the third key frame 1303. (reprojection)

Next, referring to steps 1230 and 13 of FIG. 12, the processor 170 of the autonomous vehicle calculates an error value between the projected value and the third image 1330 according to step 1220 of FIG. 12. . More specifically, the processor 170 calculates an error between the point at which the 3D landmark is projected on the third key frame 1303 and the feature point detected in the third image 1330.

Finally, referring to steps 1240 and 13 of FIG. 12, the processor 170 of the autonomous vehicle performs a process of minimizing an error value calculated according to step 1230 of FIG. 12.

In performing step 1240 of FIG. 12, the processor 170 uses wheel odometry of the autonomous vehicle to minimize an error value. Since the wheel odometer can be used to know the scale value corresponding to the position change amount of the autonomous vehicle, the autonomous vehicle according to the embodiment of the present invention can calculate the error value more accurately. The error value can be corrected more accurately.

Steps 1230 and 1240 of FIG. 12 may be understood by Equations 2 to 8 below. In Equations 2 to 4 below, R is a rotation vector set of at least one key frame 1301, 1302, and 1303 shown in FIG. 13, and t is at least one shown in FIG. A translation vector set of key frames 1301, 1302, 1303, X represents a set of positions of the 3D landmark 1350 shown in FIG. 13. N represents the number of key frames in the set, and M represents the number of 3D landmarks in the set.

Equation 5 shows the LBA minimization objective function according to the embodiment of the present invention, Equation 6 shows the reprojection error between the i and j th keyframe, Equation 7 Denotes a scale error term between the i and j th keyframes. The purpose of Equation 8 is to find a solution of R, t, and X so that the reprojection error term of Equation 6 and the magnitude error term of Equation 7 ideally have a value of zero.

는 지표 변수로서

가 i 번째 키프레임에서 관측될 경우 1을 갖고 관측되지 않을 경우 0을 갖는다. [수학식 6] 에서

와

는 각각 i 번째 키 프레임에서의 회전벡터와 이동벡터이고,

는 i 번째 키프레임에서 관측된

와 매칭된 2차원 영상 좌표이다. 또한, [수학식 6] 에서

는 3차원 점의 2차원 영상으로의 투영을 표현하는 함수이다. In Equation 5, W is an arbitrary local window size,

Is an index variable

Has 1 if it is observed in the i-th keyframe and 0 if it is not observed. In [Equation 6]

Wow

Are the rotation and movement vectors at the i-th key frame, respectively.

Is observed at the i-th keyframe

Is a two-dimensional image coordinate matched with. Also, in [Equation 6]

Is a function representing the projection of a three-dimensional point to a two-dimensional image.

은 도 13에 도시된 본 발명의 실시예에 따른 휠 오도미터로 관측한 i 번째 키프레임(1301) 및 j 번째 키프레임(1302) 사이의 실제 이동 거리(1360)를 나타낸다. 즉, 본 발명의 실시예에 따른 자율주행 차량의 프로세서(170)는 휠 오도미터를 통해 검출된 실제 이동 거리(1360)를 크기 오차항(scale error)의 보정에 이용한다.On the other hand, in [Equation 7]

Denotes the actual moving distance 1360 between the i th keyframe 1301 and the j th keyframe 1302 as viewed with the wheel odometer according to the embodiment of the present invention shown in FIG. 13. That is, the processor 170 of the autonomous vehicle according to the exemplary embodiment of the present invention uses the actual moving distance 1360 detected through the wheel odometer to correct the scale error term.

The magnitude error term of Equation 7 corrects the estimated posture to fit the measured distances 1360 and 1370 of the wheel odometer between the key frames 1301, 1302, and 1303. Since the processor 170 of the autonomous vehicle according to the embodiment of the present invention can optimize Equation 6 using the size error term, the actual size and the moving path of the 3D landmark 1350 can be more accurately estimated. That has a technical effect.

Finally, the processor 170 of the autonomous vehicle according to the embodiment of the present invention calculates the optimal R, t, and X by calculating [Equation 8] using at least one function optimization technique. The function optimization technique is, for example, the Levenberg-Marquardt method.

In summary, according to the scLBA algorithm performed by the processor of the autonomous vehicle according to the embodiment of the present invention, since the measured value obtained from the wheel odometer is used in the objective function, the actual size information of the movement path between key frames is obtained. You can nonlinearly optimize the terms you represent and the reprojection terms of the feature points. Accordingly, the scLBA algorithm performed by the processor of the autonomous vehicle according to the embodiment of the present invention can effectively solve the problem of drift caused by the LBA algorithm according to the prior art and the problem that there is no unit in the estimated path / environment map. have.

The present invention described above can be embodied as computer readable codes on a medium in which a program is recorded. The computer-readable medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include hard disk drives (HDDs), solid state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, and the like. This also includes implementations in the form of carrier waves (eg, transmission over the Internet). The computer may also include a processor or a controller. Accordingly, the above detailed description should not be construed as limiting in all aspects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

100: autonomous vehicle

Claims (10)

In autonomous vehicles,
A camera for obtaining at least two image frames of the external environment of the autonomous vehicle; And
And a processor configured to calculate a translation motion and a rotation motion of the autonomous vehicle based on the obtained at least two image frames.
The processor,
Obtain a first key frame including a first image and a second key frame including a second image through the camera, generate a 3D landmark based on feature points included in the first image and the second image, and ,
Further obtaining a third key frame including a third image through the camera, reprojecting the generated 3D landmark to the third key frame, and reconstructing the reprojected value and the third image Autonomous vehicles that calculate an error value. The method of claim 1,
The camera is provided with a plurality,
The processor,
And an image frame obtained by each of the plurality of cameras, and selecting a first camera based on the sampling result. The method of claim 2,
The processor,
Autonomous vehicle, characterized in that for generating a random sample consensus (RANSAC) model from the image frame of the selected first camera. The method of claim 3,
The processor,
Apply the generated RANSAC model to an image frame of a second camera different from the selected first camera,
And calculating a cost of the RANSAC model applied to the image frame of the second camera. The method of claim 4, wherein
The processor,
If it is determined that the calculated cost is greater than or equal to a predetermined value, the RANSAC model is applied to an image frame of a camera other than the first camera, and the translation motion and rotation motion of the autonomous vehicle are calculated. Autonomous vehicle, characterized in that. delete delete The method of claim 1,
The difference value includes a reprojection error term and a scale error term,
The processor,
Autonomous vehicle, characterized in that for correcting the reprojection error term and the size error term using a function optimization technique. The method of claim 8,
The autonomous vehicle further includes a wheel odometer,
The processor,
And controlling the wheel odometer to further detect the amount of movement of the autonomous vehicle from the first key frame to the second key frame. The method of claim 9,
The processor,
And the detected movement amount is used to correct the magnitude error term.

Images