KR102246706B1 - Autonomous driving device and method for operating autonomous driving device in anomaly situation - Google Patents

Abstract

Provided are an autonomous driving device and method for operating the same. According to one embodiment of the present invention, the autonomous driving device comprises: a camera unit acquiring information on an image on the exterior of a vehicle; a processor including at least one deep neural network to analyze the image information and outputting a result value; and an image analysis unit including a memory unit for storing the result value.

Description

Autonomous driving device and method of operating autonomous driving device in abnormal road conditions {AUTONOMOUS DRIVING DEVICE AND METHOD FOR OPERATING AUTONOMOUS DRIVING DEVICE IN ANOMALY SITUATION}

The present invention relates to an autonomous driving device and a method of operating the autonomous driving device in abnormal road conditions.

Vehicles may be classified into an internal combustion engine vehicle, an external combustion engine vehicle, a gas turbine vehicle, or an electric vehicle, depending on the type of the prime mover used.

An autonomous vehicle may refer to a vehicle that can operate by itself without a driver or by assisting a driver.

These self-driving cars are developing by combining existing automobile technology and cutting-edge IT technology. In particular, with the development of big data and deep learning technologies, research on automobiles combined with artificial intelligence is being actively conducted.

Deep learning can be defined as a set of high-level abstraction algorithms through a combination of several nonlinear transducers, and is a field of machine learning that teaches human thinking to computers.

The information on an autonomous vehicle using deep learning is disclosed in Korean Patent Registration No. 10-2026300, Korean Patent Application Publication No. 10-2018-0110566, and the like.

The problem to be solved by the present invention is to provide an autonomous driving device capable of determining whether or not there is an abnormal situation on a road.

Another problem to be solved by the present invention is to provide a method of operating an autonomous driving device capable of checking whether there is an abnormal situation on a road.

Another object of the present invention is to provide an autonomous driving device with improved autonomous driving performance.

Another problem to be solved by the present invention is to provide a method of operating an autonomous driving device with improved autonomous driving performance.

An autonomous driving device according to an embodiment of the present invention for solving the above problem includes a camera unit that obtains image information outside a vehicle, and at least one deep neural network to analyze the image information and output a result value. An image analysis unit including a processor and a memory unit for storing the result value, and an autonomous driving control unit generating a control signal based on the result value output from the image analysis unit, wherein the camera unit Acquires image information and second image information of a second viewpoint, and the processor compares the similarity between the first image information and the second image information through the deep neural network to output the result value, and the result value The abnormal situation is judged based on.

Also, the first image and the second image may be sequentially updated.

In addition, the first image may be an image of a viewpoint immediately before, and the second image may be an image of a current viewpoint.

In addition, the deep neural network includes a first sub-network for outputting a first output value, a second sub-network for outputting a second output value, a weight sharing module for sharing weights of the first sub-network and the second sub-network, and the It may refer to a network including a comparison module that compares the first output value and the second output value.

In addition, the first image may not include anomaly data, and the second image may include anomaly data.

In addition, if the processor determines that anomaly data that does not exist in the first image exists in the second image, the processor may determine this as an abnormal situation.

In addition, the memory stores a plurality of feature categories, and the processor compares the result value output in the abnormal situation with the feature category, but if the result value is not found in the feature category, a new category is created. Can be stored in the memory.

In addition, in the abnormal situation, the processor numerically outputs the risk level of the abnormal situation, and the autonomous driving controller may generate the control signal based on the numerical output value.

The method of operating an autonomous driving device according to an embodiment of the present invention for solving the above problem includes obtaining image information, processing the image information, and controlling the autonomous driving device based on the processed image information. Including, wherein the camera unit acquires first image information of a first view and second image information of a second view, and the processor is a similarity between the first image information and the second image information through the deep neural network. The result value is output by comparing and the abnormal situation is determined based on the result value.

In addition, analyzing the image information may further include quantifying and outputting a risk level of an abnormal situation.

The means for solving the problem is not limited thereto, and more specific means are dealt with in more detail in the specification.

According to embodiments of the present invention, the autonomous driving device may determine a normal situation and an abnormal situation, and perform vehicle manipulation in response thereto.

In addition, by determining normal and abnormal situations even in undefined situations, it is possible to flexibly cope with unpredictable road conditions.

The effects according to the present invention are not limited by the contents exemplified above, and more various effects are included in the present specification.

1 is a schematic diagram illustrating an autonomous driving apparatus according to an embodiment of the present invention.
2 is a block diagram illustrating an autonomous driving apparatus according to an embodiment of the present invention.
3 is a partial block diagram of an autonomous driving device according to an embodiment of the present invention.
4 is a schematic diagram showing an artificial neural network included in an embodiment of the present name.
5A is a schematic diagram for explaining an embodiment of the present invention.
5B is a block diagram showing a Siamese network included in an embodiment of the present invention.
6 is a schematic diagram schematically showing a method of determining similarity of images in an autonomous driving device according to an embodiment of the present invention.
7 is a block diagram illustrating an embodiment of the present invention.
8 is a photograph illustrating a training data set according to an embodiment of the present invention.
9 is a partial block diagram of an autonomous driving device according to an embodiment of the present invention.
10 is a block diagram illustrating a method of operating an autonomous driving device according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various forms different from each other, and only these embodiments make the disclosure of the present invention complete, and common knowledge in the technical field to which the present invention pertains. It is provided to completely inform the scope of the invention to the possessor, and the invention is only defined by the scope of the claims.

When elements or layers are referred to as “on” or “on” of another element or layer, it is possible to interpose other layers or other elements in the middle as well as directly above other elements or layers. All inclusive. On the other hand, when a component is referred to as "directly on" or "directly on", it means that no other components or layers are interposed therebetween.

The spatially relative terms "below", "beneath", "lower", "above", "on", "on", "top ( upper)" or the like may be used to easily describe a correlation between one component or components and another component or components, as shown in the drawings. Spatially relative terms should be understood as terms including different directions of a configuration during use or operation in addition to the directions shown in the drawings. For example, when the configuration shown in the drawing is reversed, the configuration described as "below" of the other configuration may be placed on the "top" of the other configuration. In addition, a configuration described as being located on the "left" side of another configuration based on the drawing may be located on the "right side" of another configuration depending on the viewpoint. Accordingly, the exemplary term “below” may include both directions below and above. The composition may be oriented in other directions, and in this case, terms that are spatially relative may be interpreted according to the orientation.

Although the first, second, and the like are used to describe various components, it goes without saying that these components are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In addition, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, one or more other features or numbers, The possibility of the presence or addition of steps, actions, components, parts, or combinations thereof is not precluded.

The same reference numerals are used for the same or similar parts throughout the specification.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

1 is a schematic diagram illustrating an autonomous driving apparatus according to an embodiment of the present invention. 2 is a block diagram illustrating an autonomous driving apparatus according to an embodiment of the present invention.

1 and 2, an autonomous driving device according to an embodiment of the present invention includes a camera unit 200, a position measurement unit 550, an image analysis unit 300, and an autonomous driving control unit 500. .

The autonomous driving apparatus according to an embodiment may include a transportation means for driving on a road or track. For example, an autonomous driving device may be understood as a concept including a car, a train, and a motorcycle. In addition, the autonomous driving apparatus may be understood as a concept including an internal combustion engine vehicle having an engine as a power source, a hybrid vehicle including an engine and an electric motor as a power source, and an electric vehicle including an electric motor as a power source.

For convenience of explanation, FIG. 1 illustrates a case where the autonomous driving device is an autonomous driving vehicle 100. However, this is only an example, and the type of the autonomous driving device is not limited anymore.

In an embodiment, the autonomous vehicle 100 may be an unmanned vehicle that operates without a driver. In another embodiment, the autonomous driving vehicle 100 may be a manned vehicle that is accompanied by a driver but assisted by an autonomous driving system. In another embodiment, in the autonomous driving vehicle 100, a driver may turn on/off an autonomous driving system. When the autonomous driving system is turned on, the autonomous driving system is operated, and when the autonomous driving system is turned off, the driver can drive directly.

In an embodiment, the autonomous vehicle 100 may be a vehicle to which various application control technologies for implementing a smart car are applied. For example, the autonomous vehicle 100 is a technology that maintains the distance between vehicles, a rear side warning system, an automatic emergency braking system, a lane departure warning system, a lane maintenance support system, Advanced Smart Cruise Control (ASCC), and congestion section support. System.

The camera unit 200 may acquire information outside the autonomous vehicle 100. In an embodiment, the camera unit 200 may include at least one camera to achieve this function. The camera may acquire image information outside or around the autonomous vehicle 100. To this end, the camera includes at least one lens, at least one image sensor, and at least one processor that is electrically connected to the image sensor and processes a received signal, and generates data on external image information based on the processed signal. Can include.

The camera unit 200 may obtain information on a road condition. As shown in FIG. 1, information such as other vehicles 91 and 92, signs 93, and natural environment 94 may be listed on the road. The camera unit 200 may acquire information listed outside the vehicle of the autonomous driving apparatus 100 (which may include all types of image information existing outside the vehicle) within the camera range CR.

In an embodiment, the camera may be at least one of a mono camera, a stereo camera, and an AVM (Around View Monitoring) camera. In an embodiment, the camera may acquire location information of an object, distance information of an object, or information of a relative speed with an object by using various image processing algorithms.

In an embodiment, the camera may be mounted in a position in which the field of view (FOV) can be secured in the autonomous vehicle 100 in order to photograph the outside of the vehicle. For example, the camera may be disposed around a front windshield, a front bumper, or a radiator grill of the autonomous vehicle 100 to acquire an image of the front of the vehicle. In addition, the camera may be disposed adjacent to the rear glass, rear bumper, trunk or tail lamp of the vehicle to acquire an image of the rear of the vehicle. In addition, the camera may be disposed adjacent to a side window, a side mirror, a fender, or a side door of the vehicle to acquire an image of the vehicle side.

In another embodiment, the camera may be placed inside a vehicle. For example, it may be disposed adjacent to the room mirror like a general black box.

The camera unit 200 may provide image information acquired from outside the vehicle to the image analysis unit 300. Specifically, the camera unit 200 may provide a continuous video or an image acquired at a constant frame rate. In an embodiment, the camera unit 200 may provide a first view point t-1 and a second view point t0 image to the image analysis unit 300. In an embodiment, the second time point t0 is the current time point, and the first time point t-1 may be a time point before the current time point, that is, a time point immediately before the current time point. The point in time immediately before may mean a point in the past that is a certain time before the present point in time. The predetermined time is not limited, and may be in seconds, minutes, or subdivided by seconds.

The position measuring unit 550 may measure the current position of the autonomous driving device. That is, it is possible to measure and provide location information of the autonomous driving device.

To this end, the position measuring unit 550 may include a position measuring means. The position measuring means may include, for example, one or more selected from the group consisting of a GPS module, an IMU module, and a Barometer module capable of receiving a position signal from at least one GPS satellite. However, this is exemplary, and the position measuring unit 550 is not limited thereto. If it is a means to check the position of the autonomous driving device, it may be adopted as the position measuring unit 550 according to an embodiment of the present invention.

In order to display the location measured by the location measurement unit 550, an embodiment of the present invention may include map data including geographic information of the country. Also, the map data may be implemented in an electronic map, a program including an electronic map, or a navigation format.

When the map data includes a program, the program may be executed by the processor 310 described later. Also, map data and the like may be stored in the memory 330.

The image analysis unit 300 may compare the image of the first viewpoint and the image of the second viewpoint. Specifically, the image analysis unit 300 may detect a meaningful change required for autonomous driving by determining the similarity between the image of the first viewpoint and the image of the second viewpoint.

Hereinafter, the image analysis unit 300 will be described in detail with reference to FIG. 3.

3 is a partial block diagram of an autonomous driving device according to an embodiment of the present invention.

The image analysis unit 300 may include a processor 310, a network module 320, and a memory 330.

Processor 310 may be configured with one or more cores, a central processing unit of a computing device (CPU, Central Processing Unit), a general-purpose graphics processing unit _(GPGPU, General Purpose Graphics Processing Unit), a tensor processing unit (TPU: Tensor Processing Unit) or a neural processing unit (NPU, Neural Processing Unit) for data analysis and deep learning. The processor 310 may read a computer program stored in the memory 330 to perform some functions of the autonomous driving apparatus according to some embodiments of the present invention.

In an embodiment, the processor 310 may perform calculations for learning a neural network. The processor 310 is a calculation for learning a neural network, such as processing input data for learning in Deep Learning, extracting features from the input data, calculating errors, and updating the weights of the neural network using backpropagation. You can do it.

In an embodiment, a plurality of processors 310 may process learning of a network function. For example, the CPU and GPGPU can work together to learn network functions and classify data using network functions.

In one embodiment, an image processed by the processor 310 using a network function is an image stored in the memory 330, an image acquired by the camera unit 200 and provided to the image analysis unit 300, and the network module 320 It may be an image provided by an external database or other computing device. In addition, the data obtained by the network module 320 may be used to train a neural network to be described later.

In an embodiment, the memory 330 may store information necessary for the image analysis unit 300 to function.

In one embodiment, the memory 330 is a volatile memory (eg, dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.)) or a non-volatile memory, for example, OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, NAND flash memory, NOR flash memory, etc.), SSD (Solid State Drive).

In an embodiment, the memory 330 may include an external memory. The external memory includes a flash drive, for example, compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (mini-SD), extreme digital (xD), or a memory stick. It may contain more.

The memory 330 may store information that is the basis for the processor 310 to perform a function. The information stored in the memory 330 may be previously input information, an image provided from an external database or another computing device by the network module 320, or an image obtained from the camera unit 200.

The network module 320 may transmit/receive information to and from other components of the autonomous driving device, another computing device, or an external server according to some embodiments of the present invention. For example, the network module 320 may receive training image data from a training image database or the like. In addition, the network module 320 may enable communication between a plurality of computing devices so that learning of a network function is distributed in each of the plurality of computing devices. In addition, the network module 320 may perform distributed processing of data classification using a network function.

In an embodiment, the processor 310 may generate location information-image information combination information by combining the location information provided by the location measurement unit 550 with the image information provided by the camera unit 200. That is, in an embodiment, the information processed by the processor 310 may be information in which image information and location information are matched.

In more detail, the location information and the image may be combined based on a viewpoint. For example, location information obtained at a first viewpoint, which is an arbitrary viewpoint, and image information obtained at the first viewpoint, may exist as data in a combined form. That is, the combined information in which the location information and the image are combined is data in a form in which the location information obtained by the location measurement unit 550 at the same time point and the image and/or image information obtained by the camera unit 200 at the same time point are combined. Can be That is, the time point corresponding to the image and/or image information may be stored as metadata.

In an embodiment, the first image 51 may be a first viewpoint t-1, that is, an image of a viewpoint immediately before. The first image 51 may be an image that serves as a basis for an anomaly determination. The second image 52 may be a second viewpoint t0, that is, an image of a current viewpoint. The first image 51 and the second image 52 may be continuously or sequentially updated as the autonomous vehicle 100 travels.

In another embodiment, the first image 51 may be an image previously stored in the memory 330. The image pre-stored in the memory 330 may include the image itself or an extracted feature. Also, the image pre-stored in the memory may be an image provided and stored from another computing device or an external server. That is, the processor 310 may determine the degree of similarity by comparing an image previously stored in the memory 330 with an image of the current viewpoint (which may be the second image 52).

As the vehicle travels, the processor 310 may combine the location information of the first view-image information and the location information of the second view-image information to generate combined information, and process the combined information. Also, the processor 310 may store the generated combination information in the memory 330.

The location information processed in this step may include information such as latitude/longitude/altitude, GPS coordinates, or a range of GPS coordinates. As described above, the processor 310 may perform machine learning. More specifically, the processor 310 is a machine learning algorithm that attempts a high level of abstraction (summarizing key contents or functions in a large amount of data or complex data) through a combination of several nonlinear transformation methods. It may include an image analysis model built using a set of deep learning.

In other words, the processor 310 may include at least one deep neural network (DNN). Hereinafter, this will be described in detail with reference to FIG. 4 and the like.

4 is a schematic diagram showing an artificial neural network included in an embodiment of the present name.

The terms are summarized for convenience of explanation. In the present specification,'neural network','network function', and'neural network' may be used with the same meaning. A neural network can be made up of a set of interconnected computational units, which can generally be referred to as'nodes'. These'nodes' may also be referred to as'neurons'. The neural network may include at least one or more nodes. Nodes constituting neural networks may be interconnected by one or more'links'.

In a neural network, one or more nodes connected through a link may relatively form a relationship between an input node and an output node. The concepts of an input node and an output node are relative, and any node in an output node relationship with respect to one node may have an input node relationship in a relationship with another node, and vice versa. As described above, the relationship between the input node and the output node can be created around a link. One or more output nodes may be connected to one input node through a link, and vice versa.

In the relationship between the input node and the output node connected through one link, the value of the output node may be determined based on data input to the input node. Here, it may have a weight that interconnects the input node and the output node. The weight may be variable, and in order for the neural network to perform a desired function, it may be changed by a user or an algorithm. For example, when one or more input nodes are interconnected to one output node by respective links, the output node includes values input to input nodes connected to the output node and a link corresponding to each input node. The output node value may be determined based on the weight.

As described above, in a neural network, one or more nodes are interconnected through one or more links to form an input node and an output node relationship in the neural network. The characteristics of the neural network may be determined according to the number of nodes and links in the neural network, an association relationship between nodes and links, and a weight value assigned to each of the links. For example, when there are the same number of nodes and links, and two neural networks having different weight values between the links exist, the two neural networks may be recognized as being different from each other.

As shown in FIG. 4, the neural network may include one or more nodes. Some of the nodes constituting the neural network may configure one layer based on distances from the initial input node. For example, a set of nodes having a distance n from an initial input node may constitute n layers. The distance from the initial input node may be defined by the minimum number of links that must be traversed to reach the corresponding node from the initial input node. However, the definition of such a layer is arbitrary for explanation, and the order of the layer in the neural network may be defined in a different way than that described above. For example, the layer of nodes may be defined by the distance from the last output node.

The first input node may mean one or more nodes to which data is directly input without going through a link in a relationship with other nodes among nodes in the neural network. Alternatively, in the relationship between nodes in a neural network network based on a link, it may mean nodes that do not have other input nodes connected by a link.

Similarly, the final output node may mean one or more nodes that do not have an output node in relation to other nodes among nodes in the neural network.

The hidden node may refer to nodes constituting a neural network other than the first input node and the last output node.

The deep neural network may mean a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Using a deep neural network, it is possible to identify the potential structure of data (Latent Structrue). In other words, it is possible to understand the potential structures of photos, texts, videos, voices, and music (for example, what objects are in photos, what are the content and emotions of the text, what are the contents and emotions of the voice, etc.). . In one embodiment, the deep neural network is a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), and a deep trust network (DBN: Deep Belief Network), a Q network, a U network, and a Siam network.

5A is a schematic diagram for explaining an embodiment of the present invention.

5A illustrates how a deep neural network (DNN) processes an image.

In an embodiment, the deep neural network (DNN) may extract each feature from the input first image 51 and the second image 52. The deep neural network (DNN) may determine the similarity between the first image 51 and the second image 52 by comparing each of the extracted features.

In an embodiment, the processor 310 may determine the similarity by obtaining a mathematical distance between the features of the first image 51 and the features of the second image 52. When the similarity between the first image 51 and the second image 52 is less than or equal to a predetermined threshold, the processor 310 converts the second image 52 into an image including anomaly data, that is, the current situation is abnormal. In other words, a result value of dissimilarity can be output.

In the present specification, the term'abnormal situation (or state)' may be used in the opposite sense of a normal situation or a general situation.

A normal situation or a general situation refers to a normal situation on the road that is generally expected when a vehicle is driving, such as a road situation on which one or more vehicles are running, a situation in which one or more vehicles are driving, a situation in which a vehicle is stopped according to a signal, etc. can do.

An abnormal situation is an unusual situation opposite to this, and may mean a situation that does not normally occur or is not expected when a vehicle is driving on the road, such as an accident situation, an unexpected situation, a road damage situation, and the appearance of animals or people. That is, the abnormal situation may mean a situation in which other actions are required (eg, avoidance, stop, etc.), not a normal driving.

Words such as'abrupt situation','abnormal situation', and'a situation in which anomaly data exists' used below may be used as an example of referring to an abnormal situation.

In another embodiment, the processor 310 compares the first image 51 and the second image 52 with a deep neural network (DNN) to express which pixel is actually different from the two images in pixel-by-pixel segmentation. You can also create an output (Pixel by Pixel segmentation output).

In another embodiment, the processor 310 may compare the outputs of the networks with an arbitrary comparison algorithm generated by the deep neural network (DNN). The processor 310 not only simply compares the difference between the two features, but also outputs the degree of the difference between the two features and the location of the pixels in the difference portion by using a deep neural network (DNN) to compare the two features. I can.

In this manner, the processor 310 may determine whether the current situation (represented by the second image 52) is compared with the first image 51 that is a normal situation to determine whether it is a normal situation or an abnormal situation. In addition, in the case of an abnormal situation, the processor 310 may determine the risk level of the anomaly data shown in the second image 52. In an embodiment, the risk level of the anomaly data may be determined by a deep neural network. To this end, a deep neural network may be trained by employing any one of the above-described methods, and the learned deep neural network may quantify and express the risk level. The output risk level may be provided to the autonomous driving control unit 500 to be described later and used as a basis for controlling the autonomous driving vehicle 100.

As described above, when an abnormal situation is determined by determining the similarity between the first image 51 acquired at the first time point t-1 and the second image 52 acquired at the second time point t0, It is possible to judge abnormal situations and respond immediately to them without the help of DB. In addition, even in a number of undefined abnormal situations that may occur on the road, the abnormal situation may be determined through similarity measurement to respond to this. That is, according to some embodiments of the present invention, not simply extracting an object from an image and defining it, but by determining the similarity between two images with different viewpoints and determining an abnormal situation, it is possible to immediately respond to an undefined abnormal situation. There is an advantage.

In addition, the first image 51 and the second image 52 may be immediately updated as the autonomous vehicle 100 operates. Accordingly, the first image 51, which is the basis for determining the abnormal situation, may be continuously updated. That is, by determining the current state based on the continuously updated normal state, it is possible to recognize an abnormal situation occurring according to the driving of the autonomous vehicle 100.

5B is a block diagram showing a Siamese network included in an embodiment of the present invention.

In an embodiment, the deep neural network may be a Siamese network.

. The Siamese network can form a cluster from the data and use it to measure the similarity between the input images. To this end, the Siamese network 400 at least partially shares two or more neural networks (the first sub-network 410 and the second sub-network 420) and a comparison module 440 that receives outputs from each of the sub-networks. ) Can be included.

Although the first sub-network 410 and the second sub-network 420 are separated for convenience of description, the configuration of the Siamese network 400 is not limited thereto. That is, in another embodiment, the first sub-network 410 and the second sub-network 420 may at least partially overlap (functionally). In another embodiment, the first sub-network 410 and the second sub-network 420 are integrally formed and functionally separated to perform the functions described below.

At least two images may be input to the Siamese network 400. The Siamese network 400 may output a result of determining the similarity of the two input images. In order to process an image, each sub-network of the Siamese network 400 may include a convolutional neural network that receives an image. Two sub-networks that receive an image from the Siamese network 400 may share at least some of the weights. That is, the sub-networks included in the Siamese network may share weights by the weight sharing module 430, and through this, the Siamese network 400 may extract and compare features with a common weight for two input data.

The Siamese network 400 is a network structure capable of learning to measure the similarity of input data, and may be used when there are many categories and insufficient data for training. The learning method of measuring the similarity of input data may include a process of finding a function that maps an input pattern to a target space such that a simple distance in the target space approaches a semantic distance in the input space.

Specifically, the learning method of measuring the similarity of input data may include a process of calculating the weight w in a function having the weight w of the sub-network. Specifically, it includes the process of calculating the weight w so that the similarity function has a small value when the input data is in the same category in the subnetwork function, and the similarity function has a large value when the input data is in different categories. can do. The weight w of each sub-network may be shared by the weight sharing module 430, respectively. The Siamese network 400 may process each input data as a function having a weight shared by the sub-network.

Convolutional neural networks included in each sub-network may share weights between nodes corresponding to each other. The similarity of features output from the convolutional neural network may be determined by the comparison module 440. In an embodiment, the comparison may be performed based on a mathematical distance of features output from both convolutional neural networks. The Siamese network 400 employing a method of comparing both images can recognize two images even when the training data is insufficient, and has general recognition performance because it is not sensitive to rotation or transformation of image data. I can. It is possible to secure the ability to classify abnormal situations.

Neural networks include supervised learning, unsupervised learning, weekly-supervised learning, semi-supervised learning, unsupervised learning, and self-supervised learning. Self-supervised learning), meta-learning, and reinforcement learning may be learned in at least one method. Learning of neural networks is to minimize output errors. Neural network learning repetitively inputs training data to the neural network, calculates the output of the neural network for the training data and the error of the target, and inputs the error of the neural network from the output layer of the neural network in a direction to reduce the error. This is a process of updating the weight of each node of the neural network by backpropagating in the layer direction.

In the case of teacher learning, learning data in which the correct answer is labeled in each learning data is used, and in the case of comparative learning, the correct answer may not be labeled in each learning data. For example, in the case of teacher learning related to data classification, the learning data may be data in which a category is labeled in each of the learning data. The labeled training data is input to the neural network, and an error may be calculated by comparing the output of the neural network with the label of the training data. The calculated error is backpropagated in the neural network in the reverse direction, and connection weights of each node of each layer of the neural network may be updated according to the backpropagation. The amount of change in the connection weight of each node to be updated may be determined according to a learning rate. The computation of the neural network for the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of repetitions of the learning cycle of the neural network. For example, a high learning rate is used in the early stages of training of a neural network, so that the neural network quickly acquires a certain level of performance to increase efficiency, and a low learning rate can be used in the later stages of training to increase accuracy.

6 is a schematic diagram schematically showing a method of determining similarity of images in an autonomous driving device according to an embodiment of the present invention.

In an embodiment, the determination of image similarity may be performed by the image analysis unit 300, more specifically, the processor 310 included in the image analysis unit 300. In an embodiment, the processor 310 may perform a neural network operation process. That is, all computational processes of image similarity comparison (ie, learning of a neural network, feature extraction, feature comparison, etc.) may be performed by the processor 310. In other words, the expression that the neural network processes data may mean a process in which the processor 310 executes the neural network to process data.

In one embodiment, the processor 310 inputs the first image 51 and the second image 52 to the Siamese network 400, and a result based on the similarity calculation result of the two images calculated in the Siamese network 400. Can be printed.

In an embodiment, the Siamese network 400 may be used for anomaly detection.

The anomalous data may mean abnormal data deviating from a normal pattern of data. In one embodiment, a situation including such anomaly data may be defined as an abnormal situation. Data may have an unstructured pattern, and anomalous data may mean data that deviates from such an unstructured pattern. For example, a general situation on a road (roads, signs, natural scenery as a background, etc.) may have an atypical pattern of a normal situation, and an anomaly situation is data that deviates from the atypical pattern of a normal situation, for example, a rockfall, It may mean the appearance of an animal blocking the road, an accident situation, a road construction situation, etc. The normal situations and abnormal situations listed above are provided as examples for convenience of description, and the present embodiment is not limited thereto.

In an embodiment, the first image 51 may be a first viewpoint t-1, that is, an image of a viewpoint immediately before. The first image 51 may be an image that serves as a basis for an anomaly determination. The second image 52 may be a second viewpoint t0, that is, an image of a current viewpoint. The first image 51 and the second image 52 may be continuously or sequentially updated as the autonomous vehicle 100 travels.

In another embodiment, the first image 51 may be an image previously stored in the memory 330. The image pre-stored in the memory 330 may include the image itself or an extracted feature. Also, the image pre-stored in the memory may be an image provided and stored from another computing device or an external server.

During autonomous driving, abnormal situations (such as situations that may harm the self-driving vehicle, such as rockfall, the appearance of animals blocking the road, accidents, road construction situations) may occur momentarily. In this case, anomaly data that is not in the first image 51 may be displayed on the second image 52 due to a time interval between the normal situation and the abnormal situation. That is, the first image 51 may be a normal state image data that does not include the anomaly data, and the second image 52 may be an abnormal state image that includes the anomaly data.

The processor 310 may extract each feature from the first image 51 and the second image 52 input to the Siamese network 400. The Siamese network 400 may compare the extracted features to determine a similarity between the first image 51 and the second image 52. In an embodiment, the processor 310 may determine the similarity by obtaining a mathematical distance between the features of the first image 51 and the features of the second image 52. When the similarity between the first image 51 and the second image 52 is less than or equal to a predetermined threshold, the processor 310 converts the second image 52 into an image including anomaly data, that is, the current situation is abnormal. You can print the result value of. In another embodiment, the processor 310 compares the first image 51 and the second image 52 with a deep neural network, and displays a pixel-by-pixel segmentation output (Pixel by Pixel segmentation output). In another embodiment, the processor 310 may compare the outputs of the sub-networks with each other using an arbitrary comparison algorithm generated in the deep neural network. The processor 310 not only simply compares the difference between the two features, but also outputs a degree of the difference between the two features, a position of a pixel in a portion with the difference, and the like, by using a neural network to compare the two features.

In this manner, the processor 310 may determine whether the current situation (represented by the second image 52) is compared with the first image 51 that is a normal situation to determine whether it is a normal situation or an abnormal situation. In addition, in the case of an abnormal situation, the processor 310 may determine the risk level of the anomaly data shown in the second image 52. In an embodiment, the risk level of the anomaly data may be determined by a deep neural network. To this end, a deep neural network may be trained by employing any of the above-described methods, and the learned deep neural network may quantify and express the risk level. The output risk level may be provided to the autonomous driving control unit 500 to be described later and used as a basis for controlling the autonomous driving vehicle 100.

As described above, when an abnormal situation is determined by determining the similarity between the first image 51 acquired at the first time point t-1 and the second image 52 acquired at the second time point t0, It is possible to judge abnormal situations and respond immediately to them without the help of DB. In addition, even in a number of undefined abnormal situations that may occur on the road, the abnormal situation may be determined through similarity measurement to respond to this. That is, according to some embodiments of the present invention, not simply extracting an object from an image and defining it, but by determining the similarity between two images with different viewpoints and determining an abnormal situation, it is possible to immediately respond to an undefined abnormal situation. There is an advantage.

In addition, the first image 51 and the second image 52 may be immediately updated as the autonomous vehicle 100 operates. Accordingly, the first image 51, which is the basis for determining the abnormal situation, may be continuously updated. That is, by determining the current state based on the continuously updated normal state, it is possible to recognize an abnormal situation occurring according to the driving of the autonomous vehicle 100.

Hereinafter, FIG. 6 will be described in more detail.

In an embodiment, the first image 51 may be input to the first sub-network 410 and the second image 52 may be input to the second sub-network 420. The processor 310 may extract features of the input image by respectively calculating the input image using each sub-network. The first sub-network 410 and the second sub-network 420 share at least some of the weights with each other, so that the comparison module 440 selects a feature with a common weight for the first image 51 and the second image 52. It can be extracted and compared. Here, the first sub-network 410 and the second sub-network 420 may include a deep neural network structure. For example, at least one of the sub-networks may have a convolutional neural network structure.

In an embodiment, the comparison module 440 may have a convolutional neural network or an arbitrary comparison algorithm structure. In an embodiment, when the comparison method outputs whether an anomaly part exists in the second image 52, the comparison module 440 may include a mathematical comparison algorithm.

The comparison module 440 may be connected to the first sub-network 410 and/or the second sub-network 420 in series. Here,'serial' may mean that at least a part of the sub-network output is input to the comparison module 440 or that a part of the sub-network may overlap with the comparison module 440. The comparison module 440 may have a configuration capable of comparing a plurality of images or features of an image, and may generate comparison information by comparing features output from each sub-network with each other.

In an embodiment, the comparison module 440 may be in the form of a function that compares the mathematical distances of two features. In another embodiment, the comparison module 440 may be configured as a deep neural network to determine the degree of similarity between two features. When the comparison module 440 is configured as a deep neural network, the comparison module 440 may calculate a feature related to image similarity from data received from the first sub-network 410 and the second sub-network 420. .

Based on the comparison result of the comparison module 440, the Siamese network 400 may output anomaly related data. The anomaly-related data may include whether an image includes anomaly data, or, when an anomaly data is included, location information of a pixel in which anomaly exists.

7 is a block diagram illustrating an embodiment of the present invention.

8 is a photograph illustrating a training data set according to an embodiment of the present invention.

7 and 8, a deep neural network (DNN) may be trained by a training data set 600.

In an embodiment, the training data set 600 may include a normal situation image 51N, an abnormal situation image 52A, and a labeling image 52L. The normal state image 51N is an image that serves as a basis for determining an anomaly of the input data, and may be an image including only normal state image data that does not include the anomaly data. The normal situation image 51N may correspond to the first image 51 of the neural network described above after learning. That is, the normal situation image 51N may have only an atypical pattern of the normal situation and may be an image without anomaly data. The abnormal situation image 52A is an image subject to an anomaly determination, and may be an image including anomaly data. (The appearance of a wild boar in FIG. 8 is an example of anomaly data.) The abnormal situation image 52A may correspond to the second image 52 of the neural network after learning. The labeling image 52L is an image obtained by labeling a portion of the abnormal situation image 52A where an anomaly exists. As an example, FIG. 8 shows a picture in which a wild boar appears, but it is a matter of course that the training data set 600 is not limited thereto.

In one embodiment, the Siamese network 400 may be trained to determine whether an anomaly data is included in the input data by learning from the normal situation image 51N, the abnormal situation image 52A, and the labeling image 52L. . That is, the normal situation image 51N, the abnormal situation image 52A, and the labeling image 52L are input to the Siamese network 400, and the Siamese network 400 has anomaly data present in the abnormal situation image 52A. You can output whether or not. The processor 310 compares the output of the Siamese network 400 with the labeling image 52L including the labeled anomaly image to obtain an error, and backpropagates the error so that the Siamese network 400 is an abnormal situation image 52A. It can be learned so that it can be determined whether or not anomaly data exists. That is, the processor 310 may train the Siamese network 400 to determine whether or not the anomaly data exists by using the labeling image 52L labeled with the anomaly data.

In another embodiment, the Siamese network 400 may be trained to determine whether an anomaly data is included in the input data by learning from the normal situation image 51N and the labeling image 52L.

In another embodiment, the Siamese network 400 may be trained to determine whether an anomaly data is included in the input data by learning from the normal situation image 51N and the abnormal situation image 52A. The normal situation image 51N and the abnormal situation image 52A are input to the Siamese network 400, and the Siamese network 400 may output data related to location information of the anomaly data from the abnormal situation image 52A. That is, the Siamese network 400 may be trained using training data that is not labeled with the anomaly data.

In an embodiment, the processor 310 may determine an abnormal situation based on the output of the Siamese network 400.

When the Siamese network 400 outputs an abnormal situation, the processor 310 may compare the feature category previously stored in the memory 330 with the output of the Siamese network 400. The feature category pre-stored in the memory 330 may be input by a user, may be determined and stored by the Siam network 400, or may be data input and stored from another computing device or server. That is, the memory 330 may categorize features extracted by the deep neural network and store them in each feature category. As described above, the extracted features may be input and stored from other computing devices or servers.

In an embodiment, when the abnormal situation output from the Siamese network 400 is not similar to the stored feature category, the processor 310 may create and store a new category.

The processor 310 may utilize the deep neural network to compare the similarity between the abnormal situation output from the Siamese network 400 and the stored feature category. That is, in substantially the same manner as described above, the similarity may be determined by comparing the stored feature category with the output of the Siamese network 400.

In this way, when the processor 310 accumulates a new category in addition to the existing feature category, it is possible to immediately and efficiently cope with various changes in the autonomous driving situation. That is, overall autonomous driving performance can be improved.

7 illustrates a learning method of the Siamese network 400, but is not limited thereto. That is, even in the case of learning of a general deep neural network (DNN) as illustrated in FIG. 5A, the above description may be applied as it is.

In addition, as described above, the first sub-network 410 and the second sub-network 420 may be separate networks performed in different computing devices, or may be functionally separated configurations while being performed in one computing device. In addition, in another embodiment, the first sub-network 410 and the second sub-network 420 may form one deep neural network (DNN).

In an embodiment, the autonomous driving device may be operated in the following manner.

In one embodiment, the deep neural network according to some embodiments of the present invention may be learned through various types of traffic accident images. In this case, the autonomous driving apparatus may determine a similarity between the learned traffic accident image and the image currently provided by the camera unit 200. As a result of the determination, the autonomous driving apparatus may determine whether a situation currently provided by the camera unit 200 is an accident situation.

When the processor 310 determines that the accident situation is based on the image provided from the camera unit 200, the processor 310 may provide the accident situation to an external server using the network module 320. In one embodiment, the external server may be a public office server such as a police station or a fire station. Through this, the external server can immediately check whether an abnormal situation occurs in a specific area in real time.

Also, the external server may be a server that collects traffic information. That is, the abnormal situation obtained through the above process is reflected in real-time traffic information and transmitted to other vehicles, thereby maintaining a smooth traffic flow.

In an embodiment, the autonomous driving device may utilize the combination information of the location information-image information described above.

In an embodiment, the image information provided by the camera unit 200 and the position information measured by the position measuring unit 550 may be combined by the processor 310. That is, the processor 310 may generate combination information of location information and image information.

In an embodiment, the processor 310 may compare a feature stored in the memory 330 with a feature extracted from a deep neural network (DNN). In addition, when the processor 310 uses the combination information, features of different viewpoints having the same location information may be compared.

That is, the processor 310 may determine whether or not the anomaly data of the image of the view B exists based on the image of the view A in which there is no anomaly data at the same first location. Accordingly, when an abnormal situation occurs in a specific region, the processor 310 may search for it in a feature category stored in the memory 330 and determine the degree of similarity. If a newly discovered abnormal situation does not exist in the previously stored feature category, the processor 310 may create a new category by storing it in the memory 330.

In addition, when the result value of the abnormal situation is output, the processor 310 may provide it to an external server using the network module 320. In one embodiment, the external server may be a public office server such as a police station or a fire station. Through this, the external server can immediately check whether an abnormal situation occurs in a specific area in real time.

Also, the external server may be a server that collects traffic information. That is, the abnormal situation obtained through the above process is reflected in real-time traffic information and transmitted to other vehicles, thereby maintaining a smooth traffic flow.

As described above, the processor 310 may determine the risk level of the abnormal situation in addition to determining the abnormal situation. For example, the processor 310 may numerically express the risk level of the abnormal situation.

In an embodiment, the determination of the level of risk may be performed by analyzing an object or anomaly data appearing in an image.

Specifically, the processor 310 may determine the risk level in consideration of the relative speed of the object or anomaly data, an expected time until collision, or an amount of impact expected when colliding with the object.

The relative speed of the object or anomaly data by the processor 310, an estimated time until collision, or an amount of impulse expected when colliding with the object may be derived by at least one neural network included in the processor 310.

That is, the processor 310 may determine the risk level using a deep neural network learned through the existing traffic accident type and the damage situation (relative speed, etc. may be considered) when a collision occurs with an object.

In one embodiment, the level of risk can be quantified. For example, a risk level of 1 to 10 or 1 to 100 may be quantified and stored in the memory 330.

In other embodiments, the level of risk may be defined in multiple stages. For example, the risk level from steps 1 to n may be quantified and stored in the memory 330.

The result value output from the image analysis unit 300 may be provided to the autonomous driving control unit 500.

9 is a partial block diagram of an autonomous driving device according to an embodiment of the present invention.

Referring to FIG. 9, the autonomous driving control unit 500 may include a control processor 510. The control processor 510 may be composed of one or more cores, a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. Processing Unit) may include a processor for data analysis and deep learning. The control processor 510 may perform calculations for learning a neural network. The control processor 510 processes input data for learning in Deep Learning, extracts features from the input data, calculates errors, and updates the weights of the neural network using backpropagation. You can perform calculations.

The control processor 510 may control the autonomous vehicle 100 based on a result value output from the image analysis unit 300. In an embodiment, the autonomous vehicle 100 may include a driving control device 700 that electrically controls various vehicle driving devices. The driving control device 700 may receive an input from the autonomous driving control unit 500 to drive the autonomous driving vehicle 100.

The drive control device 700 may include, for example, a power train 810, a brake device 820, and a steering device 830. In addition, the drive control device 700 includes a door/window drive control device (not shown), a safety device drive control device (not shown), a lamp drive control device (not shown), and an air conditioning drive control device (not shown). Not) may further include.

The power train 810 may include a power source drive control device and a transmission drive control device.

In an embodiment, the driving control device 700 may include at least one Electronic Control Unit (ECU).

The driving control device may be driven by the autonomous driving control unit 500 to be described later. Specifically, the autonomous driving control unit 500 may set a path for autonomous driving. That is, a driving plan for driving along the generated route may be generated. That is, the autonomous driving control unit 500 may provide an input signal to the driving control device 700 in order to control the movement of the vehicle according to the driving plan.

Through this process, the autonomous vehicle 100 may implement at least one Advanced Driver Assistance System (ADAS) function. ADAS includes Adaptive Cruise Control (ACC), Autonomous Emergency Braking (AEB), Forward Collision Warning (FCW), and Lane Keeping Assist (LKA). , Lane Change Assist (LCA), Target Following Assist (TFA), Blind Spot Detection (BSD), Adaptive High Beam Control System (HBA: High Beam Assist), Auto Parking System (APS), Pedestrian Collision Warning System (PDcollision Warning System), Traffic Sign Recognition (TSR), Traffic Sign Assist (TSA), Night Vision System (NV) : Night Vision), a driver status monitoring system (DSM), and a traffic jam assistance system (TJA: Traffic Jam Assist) may be implemented.

In an embodiment, when the image analysis unit 300 outputs a normal situation, the control processor 510 may execute an autonomous driving process according to a conventional manual.

In an embodiment, when the image analysis unit 300 outputs an abnormal situation, the control processor 510 may execute an autonomous driving process according to a predetermined risk level.

In an embodiment, the control processor 510 may control the autonomous vehicle 100 according to the aforementioned risk level.

For example, the control processor 510 may control the steering device and/or the brake device according to the risk level.

In a low-level danger situation, the control processor 510 provides control signals such as deceleration and gentle steering adjustment to the drive control device 700, and the drive control device 700 provides the steering device 830 and/or Alternatively, a risk may be avoided by driving the brake device 820 to decelerate or adjust the direction of the vehicle.

In a high-risk situation, the control processor 510 provides a control signal such as sudden braking and/or urgent steering control to the drive control device 700, and the drive control device 700 provides the steering device 830 and / Or by driving the brake device 820 to decelerate or adjust the direction of the vehicle to avoid the danger.

Hereinafter, a method of operating an autonomous driving device according to an embodiment of the present invention will be described.

10 is a block diagram showing a method of operating an autonomous driving device according to an embodiment of the present invention.

Referring to FIG. 10, a method of operating an autonomous driving device according to an embodiment of the present invention includes a step of acquiring image data (S100), a step of processing image data (S200), and an autonomous driving device based on the processed image data. It may include a step of controlling (S300).

The method of operating an autonomous driving device according to an exemplary embodiment may be performed by the autonomous driving device according to an exemplary embodiment of the present invention described above.

First, an operation S100 of acquiring image data may be performed. Acquisition of image data may be performed by the camera unit 200. The camera unit 200 may acquire a first image 51 of a first view point t-1 and a second image 52 of a second view point t0.

As described above, the first image 51 may be an image immediately before, and the second image 52 may be an image of a current viewpoint. As the autonomous vehicle 100 continuously operates, the first image 51 and the second image 52 may be continuously or sequentially updated.

Subsequently, an operation S200 of processing the image data may be performed. The processing of image data may be performed by the image analysis unit 300 described above. That is, the processor 310 including at least one neural network compares the first image 51 and the second image 52, extracts anomalous data from the second image 52, and outputs an abnormal situation. I can. In addition, the step of processing image data (S200) may be substantially the same as the data processing performed by the image analysis unit 300 above.

In an embodiment, the processing of image data (S200) may further include extracting a feature from the acquired image information and comparing it with the feature stored in the memory 330.

In this step, the processor 310 may determine whether the acquired image and the feature stored in the memory 330 are similar to each other to determine whether the image is in a normal or abnormal situation.

In addition, when the acquired image is new information that is not in the existing feature stored in the memory 330, the processor 310 may generate and store a new category in the memory 330.

The processing of the image data (S200) may further include combining the image information obtained by the camera unit 200 and the location information obtained by the position measuring unit 550. That is, the processor 310 may similarly determine images of two different viewpoints at the same location based on the combination information of the image information and the location information.

The images of two different viewpoints may be the first image 51 and the second image 52 sequentially acquired as described above, or may be a newly acquired image and an image previously stored in the memory 330.

In addition, in an exemplary embodiment, the processing of image data (S200) may further include providing a result of the determination to an external server.

When the processor 310 determines that the accident situation is based on the image provided from the camera unit 200, the processor 310 may provide the accident situation to an external server using the network module 320. In an embodiment, the external server may be a public office server such as a police station or a fire station. Through this, the external server can immediately check whether an abnormal situation occurs in a specific area in real time.

Also, the external server may be a server that collects traffic information. That is, the abnormal situation obtained through the above process is reflected in real-time traffic information and transmitted to other vehicles, thereby maintaining a smooth traffic flow.

Subsequently, an operation S300 of controlling the autonomous vehicle based on the processed image data may be performed. As described above, the result output from the image analysis unit 300 may be provided to the autonomous driving control unit 500. Based on the output result value, the autonomous driving controller 500 may control the autonomous driving vehicle 100. The manner in which the autonomous driving controller 500 controls the autonomous vehicle may be substantially the same as described in some embodiments of the present invention. Therefore, a detailed description thereof will be omitted.

In connection with the autonomous driving control unit 500, the method of operating the autonomous driving device in an abnormal situation according to some embodiments of the present invention may be modified as follows.

In an embodiment, the autonomous driving controller 500 may perform not only an action for safety of the own vehicle in an abnormal situation, but also an action for the safety of the entire road or the safety of other vehicles. For example, in a situation in which a person falls on the road or an animal passes by, the autonomous driving control unit 550 stops the vehicle in front of the corresponding point, turns on an emergency light to prevent other vehicles from falling into a dangerous situation, or informs of a dangerous situation in advance to prepare. can do. Recognition of various types of abnormal situations may be determined by a learned deep neural network (DNN) or a Siamese network 400. In other words, it is possible to operate the autonomous vehicle according to the situation by designating various types of abnormal situations and appropriate actions to respond to them. The learned deep neural network (DNN) or the Siamese network 400 may provide a command to the autonomous driving controller 500 to perform an appropriate action in response to an abnormal situation.

In an embodiment, the autonomous driving controller 500 may stop the vehicle in front of the abnormal situation and wait until the situation is canceled. This can prevent other vehicles from endangering due to abnormal conditions.

In addition, as described above, when the processor 310 determines an abnormal situation, it may report the abnormal situation to an external server (eg, a police station or a fire station). In addition, when the vehicle stops near the abnormal situation, the processor 310 may report the abnormal situation to an external server in real time.

When the abnormal situation is terminated, the processor 310 may provide a control signal to the autonomous driving controller 500 to perform normal driving.

In one embodiment, the deep neural network (DNN) or the Siamese network 400 learned in the above-described manner may perform similarity determination in connection with control of a vehicle led by the autonomous driving controller 500.

The deep neural network (DNN) or the Siamese network 400 is a vehicle behavior (start, stop, lane change, right turn, left turn, blinking, emergency light lighting, etc.) led by the autonomous driving controller 500 and for a certain point after the corresponding action. You can learn the situation. That is, image information on a normal situation may be collected for a certain point after the vehicle's action, and the normal situation may be learned based on this.

The definition of the normal situation learned in this way can be the basis for judging the abnormal situation.

That is, a normal situation may be learned based on the first image information acquired at a first time point after a specific action of the vehicle.

In addition, the deep neural network (DNN) or the Siamese network 400 may compare the learned first image information, that is, the normal situation information, with the image information acquired at the second view point. The first image information of the first viewpoint may be a situation expected from a viewpoint after the second viewpoint. The deep neural network (DNN) or the Siamese network 400 may determine a similarity between the second image information of the second viewpoint and the first image information of the first viewpoint predicted by learning. When it is determined that the second image information has anomalous data as a result of the similarity determination, the autonomous driving controller 500 may control the vehicle in response to an abnormal situation.

For example, when the autonomous driving controller 500 starts a vehicle from a standstill, the learned deep neural network (DNN) or the Siamese network 400 can predict a situation that normally occurs in a general stop-start situation. This prediction may be a prediction based on first image information acquired after a specific action (start here). (This first image may be learned or input in advance.

When it is determined that the deep neural network (DNN) or the Siamese network 400 has anomaly data in the second image information acquired at the second view point, the autonomous driving control unit 550 The vehicle can be controlled (assuming that the anomaly data and the corresponding correspondence of the autonomous driving controller 550 are learned as described above). In other words, you can take actions such as sudden stop and avoidance.

As described above, the autonomous driving vehicle 100 may include a form in which a driver can board and turn on/off the autonomous driving system.

In an embodiment, when the autonomous driving system is turned off, any one of the first mode and the second mode may be performed. That is, the autonomous driving apparatus according to some embodiments may perform mode switching to one of the first mode and the second mode when the autonomous driving system is turned off.

The first mode may be a mode in which the driver turns off the autonomous driving system and drives the vehicle directly, but separate learning and determination are not performed by the processor 310.

The second mode may be a mode in which the driver turns off the autonomous driving system and drives the vehicle directly, but the processor 310 learns and determines. That is, the second mode may mean a mode in which the autonomous driving system does not control the vehicle, but the learning and determination performed by the processor 310 are performed as in some embodiments of the present invention described above.

Through this, the processor 310 may obtain necessary information even when the driver directly drives. The method of obtaining information may be the same as the method described above.

In addition, the processor 310 may simulate a control method provided to the autonomous driving controller 550 in the second mode, and compare the result with the driver's driving method.

For example, in a specific situation, the processor 310 may provide a control signal to the autonomous driving controller 550 to perform a specific action. However, when the autonomous driving system is off, the control signal does not lead to direct control. However, even in this case, the processor 310 may use this as data for learning when a specific behavior scheduled in a specific situation and a driver's behavior in a specific situation are different. For example, assuming that the autonomous driving system is on, if a driver makes a stop action when a driver is expected to go straight in a specific situation, the processor 310 may store the situation and use it as data for learning. That is, it may be created as a new category and stored in the memory unit 330.

That is, when the behavior of the vehicle and the behavior of the driver that the processor 310 plans are different, the processor 310 may learn based on this and determine at least a part of the case in which the driver's behavior is different as an abnormal situation.

In this specification, the term'~ sub' or'~ module' is a concept including all units realized by hardware, units realized by software, or units partially realized by hardware and partially realized by software. Can be understood as

Further, one unit may be realized by two or more hardware and/or software, and two or more units may be realized by one hardware and/or software.

In addition,'~ part' or'~ module' is not meant to be limited to hardware or software, and'~ part' or'~ module' may be configured to be in an addressable storage medium. , It may be configured to regenerate one or more processors. Thus, for example,'~ sub' or'~ module' refers to components and processes such as software components, object-oriented software components, class components, and task components, functions, properties, procedures, and subs. It can be understood to include routines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

In addition, the components and functions provided in the'~ parts' or'~ modules' are combined or separated into a smaller number of elements and'~ parts' or'~ modules'. Can be. In addition, components and'~ unit' or'~ module' may be implemented to reproduce one or more processors (including CPU, GPU, etc.) of the device.

The embodiments of the present invention have been described above, but these are only examples and do not limit the present invention, and those of ordinary skill in the field to which the present invention pertains should not depart from the essential characteristics of the embodiments of the present invention. It will be appreciated that various modifications and applications not illustrated above are possible. For example, each component specifically shown in the embodiment of the present invention can be modified and implemented. And differences related to these modifications and applications should be construed as being included in the scope of the present invention defined in the appended claims.

When an autonomous vehicle encounters an abnormal situation from an autonomous driving perspective, the proposed anomaly detection system attempts to stop or bypass the autonomous vehicle in the safest way, and transmits the road situation to the control center.

The autonomous vehicle actively informs the outside of the vehicle of the abnormal situation while maintaining a safe distance until the corresponding abnormal symptom is resolved so that the following vehicles (including general drivers and autonomous vehicles) are not placed on the same abnormal road condition. However, abnormal road conditions are reported to public offices such as police and fire departments, and they refer to conditions that can be resolved immediately if the relevant agency is dispatched to the site.

In addition, the proposed system can be actively utilized as a method of determining an error of the vehicle control device or diagnosing a system failure when the current input image is out of the predicted value of the normal category.

100: autonomous vehicle
200: camera unit
300: image analysis unit
500: autonomous driving control unit

Claims (16)

A camera unit that acquires image information outside the vehicle;
An image analysis unit including a processor including at least one deep neural network to analyze the image information and output a result value, and a memory storing the result value; And
And an autonomous driving control unit that generates a control signal based on the result value output from the image analysis unit, wherein the camera unit acquires first image information of a first view and second image information of a second view,
The processor outputs the result value by comparing the similarity between the first image information and the second image information through the deep neural network, and determines a normal situation and an abnormal situation based on the result value,
When the similarity between the first image information and the second image information is less than or equal to a predetermined threshold, the processor determines that anomaly data that does not exist in the first image information exists in the second image information. An autonomous driving device that determines the abnormal situation. The method of claim 1,
The normal situation is an autonomous driving device including a road situation in which one or more vehicles are driving. The method of claim 1,
Further comprising a location measuring unit for measuring the location information of the vehicle,
The processor combines the first image information and the second image information provided by the camera unit with the first position information at the first viewpoint and the second position information at the second viewpoint, respectively, and stores them in the memory. But,
When the first location information and the second location information are the same,
The processor determines the abnormal situation by comparing a degree of similarity between the first image information and the second image information. The method of claim 1,
The image analysis unit further includes a network module connected to an external server,
The autonomous driving control unit, when the abnormal situation is output, stops the vehicle in front of the abnormal situation and waits, and provides the abnormal situation to the external server. The method of claim 1,
Further comprising a drive control device for controlling the vehicle based on the control signal,
The autonomous driving control unit is operated in any one mode selected from a first mode and a second mode,
When the processor determines the abnormal situation in the first mode, the drive control device controls the vehicle,
When the processor determines the abnormal situation in the second mode, the driving control device does not control the vehicle. The method of claim 2,
The normal situation is defined as further including a situation in which the vehicle goes straight, a situation in which it turns right, a situation in which the vehicle is turning left, and a situation in which the vehicle stops according to a signal,
The abnormal situation is an autonomous driving device defined as including an accident situation, an unexpected situation, a road damage situation, and a situation in which animals or people appear. delete delete The method of claim 1,
The memory stores a plurality of feature categories, and the processor compares the result value output in the abnormal situation with the feature category. Autonomous driving device that stores in memory. The method of claim 1,
In the abnormal situation, the processor digitizes and outputs the risk level of the abnormal situation, and the autonomous driving controller generates the control signal based on the digitized output value. The method of claim 1,
Further comprising a location measuring unit for measuring the location information of the vehicle,
The processor generates combined information by combining the location information and the image information,
The processor compares the similarity between the first image information and the second image information having the same location information. A camera unit that acquires image information outside the vehicle;
A position measuring unit that measures position information of the vehicle;
An image analysis unit including a processor including at least one deep neural network to analyze the image information and output a result value, and a memory storing the result value; And
A method for controlling an autonomous vehicle comprising an autonomous driving control unit that generates a control signal based on the result value output from the image analysis unit,
Obtaining the image information, processing the image information, and controlling an autonomous driving device based on the processed image information,
The camera unit acquires first image information of a first view and second image information of a second view,
The processor compares the similarity between the first image information and the second image information through the deep neural network, outputs the result value, and determines a normal situation and an abnormal situation based on the result value,
When the similarity between the first image information and the second image information is less than or equal to a predetermined threshold, the processor determines that anomaly data that does not exist in the first image information exists in the second image information. An autonomous driving device operating method that determines the abnormal situation. The method of claim 12,
The normal situation is a method of operating an autonomous driving device including a road situation on which one or more vehicles are running. The method of claim 12,
In the processing of the image information, the processor converts the first image information and the second image information provided from the camera unit into first position information at the first viewpoint and second position information at the second viewpoint. And storing them in the memory by combining them with each other,
When the first location information and the second location information are the same,
The processor is a method of operating an autonomous driving device for determining the abnormal situation by comparing a degree of similarity between the first image information and the second image information. The method of claim 12,
The image analysis unit further includes a network module connected to an external server,
The processing of the image information further comprises: when the abnormal situation is output, stopping the vehicle in front of the abnormal situation and waiting, and providing the abnormal situation to the external server.

delete

Images