KR20220115687A - Pedestrian communication system for self driving cars - Google Patents

Abstract

The present invention relates to a pedestrian communication system of an autonomous vehicle, in which an autonomous vehicle recognizes a pedestrian and predicts a behavior of the pedestrian. The pedestrian communication system of an autonomous vehicle according to one aspect of the present invention comprises: a detection unit recognizing a pedestrian; and an output unit displaying that the pedestrian is recognized to be viewed from the outside of the vehicle.

Description

Pedestrian communication system for autonomous vehicles

The present invention relates to a pedestrian communication system, and more particularly, to a pedestrian communication system of an autonomous vehicle between an autonomous vehicle and a pedestrian.

An automobile may be classified into an internal combustion engine automobile, an external combustion engine automobile, a gas turbine automobile, an electric vehicle, or the like, according to a type of a prime mover used.

An autonomous vehicle is a vehicle that can drive automatically without human driving. Autonomous vehicles drive autonomously by recognizing their surroundings with radar, light detection and ranging (LIDAR), GPS, and cameras and specifying a destination. Unmanned vehicles that have already been put into practical use include unmanned vehicles that patrol preset routes operated by the Israeli military, and unmanned operation systems such as dump trucks operated in overseas mines and construction sites.

The first core technologies of these autonomous vehicles are the driverless vehicle system and the actual system. In addition to simulation in the laboratory, it is a technology that actually builds an unmanned vehicle system. It implements the accelerator, reducer, and steering device, which are driving devices, for unmanned operation, and enables control using the computer, software and hardware installed in the unmanned vehicle. do.

The second core technology is to receive and process visual information using vision and sensors. As the basis of autonomous driving for unmanned driving, it is a technology that accepts image information and extracts necessary information from this image. It uses CCD (charge-coupled device) cameras as well as ultrasonic sensors and range filters to fuse information necessary for distance and driving, so that it can avoid obstacles and cope with unexpected situations through analysis and processing.

The third core technology is the integrated control system and operation monitoring fault diagnosis system technology. This technology establishes a driving monitoring system that monitors vehicle operation and gives appropriate commands according to changing situations, and provides appropriate information to the operator by analyzing various situations occurring from individual processors and sensors to diagnose system failures. or to perform the function of notifying an alarm.

The fourth core technology is intelligent control and intelligent operation devices. This technology is an unmanned driving technique that generates control commands based on mathematical analysis using real vehicle models. Intelligent Forward Control is a system that maintains the distance from the vehicle in front or obstacles by adjusting the speed by itself without the driver operating the pedals based on radar guide technology. When the driver inputs the distance to the vehicle in front, the long-range radar attached to the front of the vehicle detects the position of the vehicle in front, maintains a constant speed, decelerates, accelerates, and stops completely if necessary, which is useful in weather where visibility is difficult.

The fifth applied technology is the lane departure prevention system. This is a technology that notifies the driver of an unintentional departure situation by detecting a lane with a camera installed inside.

The sixth application technology is the parking assist system. This is a system that helps the car park in reverse line by adjusting the steering system when the driver presses the brake pedal after searching for the assist button and inserts the reverse gear. Based on infrastructure as well as vehicle-mounted sensors, the vehicle is automatically guided from the departure point to the parking space, saving unnecessary time and energy when parking, thereby minimizing cost and environmental pollution.

The seventh application technology is the automatic parking system. This is a technology in which the driver stops the car in front of the parking lot, turns off the engine, gets off, and presses the remote control lock switch twice in succession. to be.

The eighth application technology is a blind spot information guidance system. It is used to change lanes by checking obstacles and vehicles on both sides in complex road conditions, as sensors mounted on both sides of the car determine whether there are other vehicles in the blind spot not seen by the side mirrors and warn the driver. .

The biggest advantage of driverless cars is that they help fuel efficiency by avoiding repeated stops by making the control device more even because the driving speed and traffic management data match. will be. In addition, it has the advantages of solving fatigue caused by long-term driving, greatly reducing the risk of traffic accidents, speeding up road traffic flow, and reducing traffic congestion.

Furthermore, autonomous vehicles are expected to solve a number of parking-related problems by not occupying a parking space unnecessarily in the absence of a person. In particular, when an accident occurs, the autonomous vehicle may move smartly and play a role of easily assisting in handling an accident.

However, there is a problem in that pedestrians may have a vague fear of safety that the autonomous vehicle is not controlled because the discussion on the additional user experience using the autonomous vehicle has not been actively conducted yet. Therefore, it is time for a detailed discussion on communication methods for safety between autonomous vehicles and pedestrians.

Korean Patent Publication No. 2019-0049221 (2019.05.09.)

SUMMARY OF THE INVENTION The present invention is to solve the problems of the prior art, and an object of the present invention is to provide a pedestrian communication system for an autonomous driving vehicle that allows the autonomous driving vehicle to recognize a pedestrian and predict a pedestrian's behavior.

A pedestrian communication system for an autonomous vehicle according to an aspect of the present invention includes a sensing unit for recognizing a pedestrian and an output unit for displaying the recognition of the pedestrian to be visible from the outside of the vehicle.

In this case, the sensing unit may recognize the gaze of the pedestrian, and the output unit may output a cursor according to the gaze direction recognition.

In addition, the output unit may output an input menu, but may output a cursor according to gaze direction recognition on the input menu.

In addition, the pedestrian is composed of at least one, and when the sensing unit does not recognize the pedestrian's gaze, the output unit may output an alarm.

In addition, the movement of the cursor may be made according to at least one of the pedestrian's gaze, the pedestrian's motion, and the voice.

According to the present invention, the autonomous driving vehicle recognizes the pedestrian's gaze, outputs a menu screen according to the pedestrian's gaze, and then allows the pedestrian to select it, so the autonomous driving vehicle can clearly determine the pedestrian's behavior and respond to it.

1 is a block diagram of a pedestrian communication system of an autonomous vehicle according to an embodiment of the present invention.
FIG. 2 is a configuration diagram illustrating the sensing unit of FIG. 1 in more detail.
FIG. 3 is a configuration diagram illustrating the output unit of FIG. 1 in more detail.
FIG. 4 is a configuration diagram illustrating the control unit in FIG. 1 in more detail.
5 to 8 are views for explaining the operation of the pedestrian communication system of the autonomous vehicle according to an embodiment of the present invention.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numbers regardless of reference numerals, and redundant description thereof will be omitted.

The suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves.

In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in the present specification, the detailed description thereof will be omitted.

In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including ordinal numbers such as first, second, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as “comprises” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, the pedestrian communication system 1000 of the autonomous vehicle according to an embodiment of the present invention will be described. 1 is a configuration diagram of a pedestrian communication system of an autonomous vehicle according to an embodiment of the present invention, FIG. 2 is a configuration diagram illustrating the sensing unit in FIG. 1 in more detail, and FIG. 3 is a diagram showing the output unit in FIG. 1 more It is a configuration diagram showing in detail, and FIG. 4 is a configuration diagram showing the control unit in FIG. 1 in more detail.

Referring to the drawings, a pedestrian communication system 1000 of an autonomous vehicle according to an embodiment of the present invention includes an autonomous vehicle 100 and a server 200 that manages routes and performs remote control. In this case, the autonomous vehicle 100 includes an input unit 110 , a sensing unit 120 , an output unit 130 , a control unit 140 , a communication unit 150 , and a driving unit 160 .

The input unit 110 is a device for receiving a user input for driving. In the manual mode, the input unit of the autonomous vehicle 100 may include a steering input device, an acceleration input device, and a brake input device. In addition, the input unit 110 serves to input the destination of the autonomous vehicle, receives information from the server 200 according to the input destination, and continuously sets a route.

The sensing unit 120 may include a radar 121 and a lidar 122 , and may further include a camera 123 . In this embodiment, a pedestrian, a pedestrian's gaze, a pedestrian's movement, a pedestrian's movement direction, and a pedestrian's gesture are detected using the radar 121, the lidar 122, and the camera 123.

First, the radar 121 may generate information on an object outside the autonomous vehicle 100 using radio waves. The radar 121 may include at least one processor that is electrically connected to an electromagnetic wave transmitter, an electromagnetic wave receiver, and an electromagnetic wave transmitter and an electromagnetic wave receiver, processes a received signal, and generates data for an object based on the processed signal. have.

The radar 121 may be implemented in a pulse radar method or a continuous wave radar method in view of a radio wave emission principle. The radar 121 may be implemented in a frequency modulated continuous wave (FMCW) method or a frequency shift keyong (FSK) method according to a signal waveform among continuous wave radar methods. The radar 121 detects an object based on a time of flight (TOF) method or a phase-shift method using an electromagnetic wave as a medium, and calculates the position of the detected object, the distance from the detected object, and the relative speed. can be detected. In this case, the radar 121 may be disposed at an appropriate location outside the vehicle in order to detect an object located in the front, rear, or side of the vehicle.

Next, the lidar 122 may generate information about an object including a pedestrian outside the autonomous vehicle 100 by using the laser light. The lidar 122 is electrically connected to the light transmitter (not shown), the light receiver (not shown), and the light transmitter and the light receiver, processes the received signal, and generates data for the object based on the processed signal. It may include at least one processor.

The lidar 122 may be implemented in a time of flight (TOF) method or a phase-shift method. The lidar 122 may be implemented as a driving type or a non-driven type. When the lidar 122 is implemented as a driving type, the lidar 122 is rotated by a motor and can detect objects such as pedestrians around the autonomous vehicle 100 . have. When implemented as a non-driven type, the lidar 122 may detect an object located within a predetermined range with respect to the vehicle by light steering. The autonomous vehicle 100 may include a plurality of non-driven lidar.

The lidar 122 detects an object based on a time of flight (TOF) method or a phase-shift method using a laser light medium, and calculates a position of the detected object, a distance from the detected object, and a relative speed. can be detected. In this case, the lidar 122 may be disposed at an appropriate position outside the vehicle in order to detect an object located in the front, rear or side of the vehicle.

Meanwhile, the camera 123 may generate information on an object such as a pedestrian outside the autonomous driving vehicle 100 by using the image. The camera 123 may include at least one lens, at least one image sensor, and at least one processor that is electrically connected to the image sensor to process a received signal, and generate data about the object based on the processed signal. can

The camera 123 may be at least one of a mono camera, a stereo camera, and an Around View Monitoring (AVM) camera. The camera 123 may obtain position information of an object, distance information from an object, or relative speed information with an object by using various image processing algorithms.

For example, the camera 123 may acquire distance information and relative velocity information from an object based on a change in the size of the object over time in the acquired image. Also, the camera 123 may acquire distance information and relative speed information with respect to an object through a pinhole model, road surface profiling, or the like.

Also, the camera 123 may acquire distance information and relative velocity information from an object based on disparity information in a stereo image obtained from the stereo camera. The camera 123 may be mounted at a position where a field of view (FOV) can be secured in the vehicle to photograph the outside of the vehicle.

The camera 123 may be disposed adjacent to the front windshield in the interior of the vehicle to acquire an image of the front of the vehicle. Furthermore, the camera 123 may be disposed around the front bumper or radiator grill. The camera 123 may be disposed adjacent to the rear glass in the interior of the vehicle to acquire an image of the rear of the vehicle. In this case, the camera 123 may be disposed around the rear bumper, the trunk, or the tailgate. In order for the camera 123 to acquire an image of the side of the vehicle, it may be disposed adjacent to at least one of the side windows in the interior of the vehicle. Alternatively, the camera 123 may be disposed around a side mirror, a fender, or a door.

In addition, in this embodiment, the camera 123 recognizes the gaze of the pedestrian. In this case, the camera 123 produces a high-quality eye image by pan-tilt-focusing with a wide view field camera for finding the position of the eye in a wide area and the position of the found eye. It may consist of a narrow view field camera for acquisition.

In addition, the camera 123 may perform calibration and ractification of the stereo camera in order to track the gaze of the pedestrian, perform calibration between the stereo camera and the narrow-angle camera, and perform user calibration. In this case, when each calibration is completed, images may be received from the stereo camera, and a face region and an eye region may be detected from each of the input images.

Then, based on the detected 2D coordinates of the eye and the user's position information obtained through calibration, 3D position information for the pedestrian's eye area is calculated, and based on this, panning and tilting for the narrow angle camera is performed. ), auto-focus is performed, and a high-quality eye image is obtained through this. Then, the pupil center is detected from the obtained high-quality eye image, and the corneal reflection light reflected by the infrared light is detected. Through this process, the gaze position and movement of the pedestrian can be calculated from the information on the pupil center and the corneal reflected light detected through this process. In addition, when the gaze of the pedestrian is moved, it is possible to select a menu output by an output unit to be described later, such as a mouse cursor.

In addition, since the sensing unit 120 needs to utilize location information of the autonomous vehicle, it essentially further includes a GPS 124 . The GPS 124 may include at least one of a general global positioning system (GPS) and a differential global positioning system (DGPS) to generate location data of the autonomous vehicle 100 . Position data of the autonomous vehicle 10 may be generated based on a signal generated by at least one of the GPS and DGPS.

In this case, the GPS 124 may correct the location data based on at least one of an Inertial Measurement Unit (IMU) and the camera 123 of the sensing unit 120 . Also, the GPS 124 may be referred to as a Global Navigation Satellite System (GNSS).

On the other hand, the sensing unit 120 may further include a microphone 125 in order for the passenger to intervene in the autonomous driving in addition to the guardian by voice or the like even if the passenger does not perform the steering operation. Furthermore, the biometric information sensor 126 not only prepares for an emergency by sensing the passenger's heartbeat, pressure, brain waves, etc., but additionally senses fingerprint and iris information when entering and exiting the vehicle to prevent erroneous boarding. can By utilizing the biometric information sensor 126 , the autonomous vehicle 100 may detect that a passenger has boarded or alighted.

Meanwhile, the output unit 130 is normally disposed inside or outside the autonomous vehicle 100 to display driving-related situations. Furthermore, any one of a shape, a shape, and a color may be displayed on the exterior of the vehicle to provide predictability to the vehicle owner and surrounding people.

Furthermore, the output unit 130 according to an embodiment of the present invention comprises a gaze output module 131 , a menu output module 132 and an alarming output module 132 .

The gaze output module 131 outputs by displaying or alarming that the pedestrian is recognized first. At this time, the gaze output module 131 further outputs the detected gaze of the pedestrian in the form of a cursor. In addition, when the gaze of the pedestrian is moved, the gaze output module 131 causes the cursor to be moved correspondingly to be output.

On the other hand, the menu output module 132 outputs a menu asking the intention of the pedestrian (for example, a menu for whether to cross the road or not), and the gaze output module 131 outputs the cursor on the menu. . However, at this time, since the cursor can be moved according to the movement of the gaze, a menu selection is made possible only by moving and fixing the gaze. In addition, the selection of the menu at this time not only fixes the gaze of the pedestrian, but also collects the motion or voice of the pedestrian to select the menu.

For example, when the vehicle 100 detects a pedestrian and outputs a menu screen consisting of "Yes" / "No" with the query word "Do you want to cross the road?", the user's gaze is also displayed on the menu screen. After output, when the user's gaze is fixed on "yes", nods his head, or says "yes", it is accurately determined that the intention of the pedestrian is crossing the road.

However, since the pedestrian may not recognize the gaze for other purposes, in this case, the alarming output module 133 alerts the pedestrian that the gaze is not recognized, and the gaze of the pedestrian looking at this is detected. In this case, when there are a plurality of pedestrians, it may be preferable to determine the opinion of the whole when the gaze of any one is detected.

Meanwhile, the control unit 140 may be configured as a main ECU, and controls the driving unit 160 of the autonomous vehicle 100 .

In particular, the controller 140 may include a power train drive control device, a chassis drive control device, a door/window drive control device, a safety device drive control device, a lamp drive control device, and an air conditioning drive control device. The power train drive control device may include a power source drive control device and a transmission drive control device. The chassis drive control device may include a steering drive control device, a brake drive control device, and a suspension drive control device. Meanwhile, the safety device drive control device may include a safety belt drive control device for seat belt control.

The controller 140 includes at least one electronic control device (eg, a control ECU (Electronic Control Unit)). In particular, on the basis of the received signal, it is possible to control the vehicle drive unit. For example, the controller 140 may control a power train, a steering device, and a brake device based on a signal received from the sensor 120 .

Furthermore, the control unit 140 includes a pedestrian recognition module 141 , a pedestrian selection confirmation module 142 , and an output control module 143 .

The pedestrian determination module 141 detects the movement of pedestrians and pedestrians, receives signals detected by the radar 121 and the rider 122 to classify information, and the camera 123 converts the color information input as an image into black and white. convert At this time, the lidar 122 converts the received information into a coordinate plane, and the camera 123 normalizes the histogram (frequency distribution table) from the converted information.

In addition, an ROI (region of interest) is set through the coordinate system converted by the lidar 122 and the histogram obtained from the camera. Then, the ROI obtained in the setting step is converted to HOG to obtain pedestrian recognition information, and the lidar 122 obtains a distance value from the information transmitted as an RF signal. Thereafter, two pieces of information obtained by measuring the area of interest and the distance value are combined, and a pedestrian or non-pedestrian who wants to cross the road is distinguished from the combined information through SVM classification.

Pedestrians are classified through the classification of the classification step, and information is stored in the case of pedestrians. If pedestrians can be classified, the normalization step is additionally returned. If not, the algorithm is terminated. Accordingly, as a result, the autonomous vehicle 100 operates the brake while driving slowly.

At this time, the user's gaze is recognized and the menu screen is output to the outside, and the pedestrian selection confirmation module 142 finally determines which area of the menu screen the user has selected. In this case, when the user performs a voice input as described above, the selection is immediately confirmed. However, when the user's gaze direction is unclear, the output unit may request to accurately fix the gaze by causing the alarm again. Furthermore, the pedestrian selection confirmation module 142 determines that the user's gaze is fixed to any one of the menus, and when a detection such as a nod of the head is input here, the gaze is confirmed and finally the menu is selected. have. On the other hand, the output control module 143 controls the output according to the first pedestrian recognition, the cursor display output along the line of sight direction, and the general information output inside and outside the vehicle.

Meanwhile, the communication unit 150 may exchange signals with devices located outside the autonomous vehicle 100 , and may exchange signals with at least one of an infrastructure (eg, a server, a broadcasting station), another vehicle, and a terminal. have. The communication unit 150 may include at least one of a transmit antenna, a receive antenna, a radio frequency (RF) circuit capable of implementing various communication protocols, and an RF element to perform communication. For example, the communication apparatus may exchange a signal with an external device based on C-V2X (Cellular V2X) technology. In addition, the communication unit 150 is based on the IEEE 802.11p PHY / MAC layer technology and the IEEE 1609 Network / Transport layer technology based on DSRC (Dedicated Short Range Communications) technology or WAVE (Wireless Access in Vehicular Environment) standard based on the external device and signal can be exchanged for

Hereinafter, the operation of the present invention will be exemplified and described. 5 to 8 are diagrams illustrating the operation of a pedestrian communication system of an autonomous vehicle according to an embodiment of the present invention.

First, referring to FIG. 5 , it is shown that the autonomous driving vehicle that has recognized the pedestrian outputs a mark indicating that the pedestrian has been recognized as text. In FIG. 6 , the gaze of the pedestrian is tracked on the front display and displayed as a cursor (red circle), and the cursor is output on the output menu screen. Accordingly, the user can confirm the intention to pass by fixing his or her gaze on any one of yes/no menus, or further clearly input the motion or voice to confirm the intention more clearly.

7 is a diagram illustrating a case where a pedestrian is a plurality. Here, the vehicle that has detected one or more of the plurality of pedestrians first outputs a mark indicating that it has been detected, and when the gaze of any one of the pedestrians is detected, it becomes possible for all pedestrians to cross the road by selecting the menu of that one.

As such, those skilled in the art to which the present invention pertains will understand that the present invention may be embodied in other specific forms without changing the technical spirit or essential characteristics thereof.

In addition, the present specification and drawings have been disclosed with respect to preferred embodiments of the present invention, and although specific terms are used, these are only used in a general sense to easily explain the technical content of the present invention and help the understanding of the present invention, It is not intended to limit the scope of the invention. It will be apparent to those of ordinary skill in the art to which the present invention pertains that other modifications based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein.

1000: pedestrian communication system of autonomous vehicle
100: autonomous vehicle
110: input unit
120: detection unit
130: output unit
140: control unit
150: communication department
160: driving unit
200: server

Claims (5)

a sensing unit for recognizing a pedestrian; and
an output unit for displaying that the pedestrian has been recognized to be visible from the outside of the vehicle;
A pedestrian communication system of an autonomous vehicle comprising a.
According to claim 1,
The sensing unit recognizes the gaze of the pedestrian,
The output unit outputs a cursor according to gaze direction recognition.
According to claim 1,
The output unit outputs an input menu, and outputs a cursor according to gaze direction recognition on the input menu.
According to claim 1,
The pedestrian consists of at least one,
The pedestrian communication system system of an autonomous vehicle, characterized in that the output unit outputs an alarm when the sensing unit does not recognize the gaze of the pedestrian.
4. The method of claim 3,
The pedestrian communication system of the autonomous driving vehicle, characterized in that the movement of the cursor is made according to at least one of the pedestrian's gaze, the pedestrian's motion, and the voice.

Images