KR20230011775A - Method and system for detecting abnormality of object detection with vehicle - Google Patents

Abstract

Introduced are an object recognition abnormality sensing method of a vehicle and an object recognition abnormality sensing system of the vehicle. The object recognition abnormality sensing method of the vehicle of the present invention comprises the following steps of: receiving input data about a condition of vehicle interior and exterior from an input unit; generating object data which is the data about an object included in the input data based on the input data and a plurality of object recognition models in a processing unit; and extracting the object data about the object identified as the same among the object data in an abnormality sensing unit, and calculating an abnormality value by comparing the extracted object data. According to the object recognition abnormality sensing method of the vehicle and the abnormality sensing system, a failure of an object recognition apparatus can be automatically and quickly diagnosed.

Description

Vehicle object recognition anomaly detection method and anomaly detection system {METHOD AND SYSTEM FOR DETECTING ABNORMALITY OF OBJECT DETECTION WITH VEHICLE}

The present invention relates to a method and an anomaly detection system for object recognition of a vehicle, and more particularly, to a deep learning-specific IP (Intellectual Property: logic circuit block) to determine whether a deep learning-based object recognition device is abnormal. A vehicle object recognition anomaly detection method and anomaly detection system that promotes the functional safety of object recognition devices and automatically and quickly diagnoses the failure of object recognition devices by comparing with inference results through traditional algorithms of other IPs other than algorithms It is about.

Recently, mobility, which is a means of transportation including vehicles and applications, is equipped with an Advanced Driver Assistance System (ADAS) or an autonomous driving system. These systems recognize patterns by analyzing data measured through mobility external measurement sensors and images and voices such as cameras. In this process, a large amount of data is processed serially or in parallel in each device in real time. Therefore, a recognition error in one of these devices may cause a recognition error in the entire system, which greatly damages the reliability of the system. Therefore, a device that recognizes a pattern needs a technique capable of detecting and restoring such a recognition error. Through this technology, the system can ensure robustness, that is, the correct operation of the system.

However, existing devices that monitor and repair recognition errors have to turn off the power of the recognition device and diagnose which part is faulty for each controller constituting the recognition device. And, as such a diagnosis method, it is possible to determine whether or not there is a failure by measuring the voltage measured by applying an input voltage to each device, or to detect it using software such as a program that determines bad sectors. In this case, if there is no other recognition device that can replace the existing recognition device or if it is out of order, such a system may cause danger to the user in the driving environment of mobility. In addition, when a lot of resources are invested in the recognition itself, there is a concern that control may be delayed due to the input of resources even in fault diagnosis. Therefore, various techniques to solve these problems are under study.

The matters described as the background art are only for improving understanding of the background of the present invention, and should not be taken as an admission that they correspond to prior art already known to those skilled in the art.

KR 10-1786237 B1

The present invention has been proposed to solve this problem, and in order to determine whether a deep learning-based object recognition device is abnormal, it is possible to use a traditional algorithm of an IP other than an algorithm of an IP (Intellectual Property: Logic Circuit Block) specialized for deep learning. By comparing with the inference result, it is intended to promote the functional safety of the object recognition device and to provide an object recognition anomaly detection method and an anomaly detection system of a vehicle that can diagnose the failure of the object recognition device automatically and quickly.

A method for detecting an object recognition abnormality of a vehicle according to the present invention includes the steps of receiving input data about conditions inside and outside the vehicle from an input unit; generating object data that is data about an object included in the input data based on the input data and a plurality of object recognition models in a processing unit; and extracting object data related to objects identified as the same among the object data in the anomaly detecting unit, and calculating an anomaly value by comparing the extracted object data.

In the calculating step, an ideal value may be calculated as a difference between object data belonging to the same class among a plurality of object data extracted by the control unit.

After the calculating step, the control unit may control the processing unit to stop generating inference data about the object when the abnormal value is greater than or equal to the first reference value.

In the controlling step, when an ideal value continuously calculated according to time is greater than or equal to a first reference value for a first time period, the first processing unit may be controlled to stop generating inference data about the object.

In the receiving step, the input data may be image data including objects inside and outside the vehicle.

In the step of receiving the input, the input data may be sensing data of a sensor measured inside or outside the vehicle or processed data processed by autonomous driving of the vehicle with respect to the sensing data.

Further comprising the step of storing the sensing data of the sensor measured inside or outside the vehicle in the storage unit prior to the step of receiving the input, and in the step of generating the input data and the input data based on the sensing data and a plurality of object recognition models. Object data, which is data about an object included in the sensing data, may be generated.

In the generating step, the plurality of object recognition models may include an object recognition model based on deep learning.

In the generating step, a first processing unit among a plurality of processing units generates object data based on an object recognition model based on deep learning, and a second processing unit generates object data based on an object recognition model based on an algorithm other than deep learning. data can be generated.

Prior to the generating step, the control unit further comprising selecting a second processing unit to generate object data based on an object recognition model based on an algorithm other than deep learning among a plurality of processing units, and selecting the first processing unit. In the control unit, the processing unit with the lowest usage rate may be selected as the second processing unit.

The object recognition anomaly detection system of a vehicle according to the present invention,

an input unit that receives input data about conditions inside and outside the vehicle; a processing unit that generates object data, which is data about an object included in the input data, based on the input data and a plurality of object recognition models; and an anomaly detecting unit extracting object data related to an object identified as the same among object data and calculating an anomaly value by comparing the extracted object data.

The anomaly detection unit may calculate an anomaly value as a difference between object data belonging to the same class among a plurality of extracted object data.

When the abnormal value is equal to or greater than the first reference value, the controller may further include a control unit that controls the processing unit to stop generating inference data about the object.

The plurality of object recognition models include an object recognition model based on deep learning,

Among the plurality of processing units, the first processing unit generates object data based on an object recognition model based on deep learning, and the second processing unit generates object data based on an object recognition model based on an algorithm other than deep learning. can

Further comprising a control unit for processing to select a second processing unit to generate object data based on an object recognition model based on an algorithm other than deep learning among the plurality of processing units, wherein the control unit assigns the processing unit with the lowest usage rate to the second processing unit. You can choose.

According to the anomaly reasoning determination apparatus and determination method of the present invention, the present invention is not an algorithm of an IP (Intellectual Property: logic circuit block) specialized for deep learning to determine whether a deep learning-based object recognition apparatus makes an anomaly inference. By comparing with the inference result through the traditional algorithm, it promotes the functional safety of the object recognition device and has the effect of automatically and quickly diagnosing the failure of the object recognition device.

1 is a block diagram of an object recognition anomaly detection system of a vehicle according to an embodiment of the present invention.
2 is a block diagram of an object recognition anomaly detection system of a vehicle according to an embodiment of the present invention.
3 is a view showing a mechanism of an object recognition anomaly detection system of a vehicle according to an embodiment of the present invention.
4 is a flowchart of a method for detecting an abnormality in object recognition of a vehicle according to an embodiment of the present invention.
5 is a flowchart illustrating an anomaly detection mechanism of a method for detecting an anomaly in object recognition of a vehicle according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments of the present invention disclosed in this specification or application are merely exemplified for the purpose of explaining the embodiments according to the present invention, and the embodiments according to the present invention may be implemented in various forms. and should not be construed as being limited to the embodiments described in this specification or application.

In this specification or application, “includes” means that features, numbers, steps, operations, components, parts, or combinations thereof described in the specification may exist, and indicates the existence or addition of one or more other features. It should be understood that it is not excluded in advance. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an object recognition anomaly detection system of a vehicle according to an embodiment of the present invention, FIG. 2 is a configuration diagram of an object recognition anomaly detection system of a vehicle according to an embodiment of the present invention, and FIG. It is a diagram showing the mechanism of a vehicle object recognition anomaly detection system according to an embodiment of the present invention, Figure 4 is a flow chart of a vehicle object recognition anomaly detection method according to an embodiment of the present invention, Figure 5 is one of the present invention It is a flow chart showing the anomaly detection mechanism of the object recognition anomaly detection method of a vehicle according to the embodiment.

1 is a block diagram of an object recognition anomaly detection system of a vehicle according to an embodiment of the present invention, and the object recognition anomaly detection system of a vehicle according to an embodiment of the present invention inputs input data related to situations inside and outside the vehicle. receiving input unit (I); Processors O1 and O2 that generate object data, which is data about objects included in the input data, based on the input data and a plurality of object recognition models; and an anomaly detecting unit (A) that extracts object data for objects identified as the same among the object data and compares the extracted object data to calculate an anomaly value.

As shown in FIG. 1, the input unit (I) receives input data related to conditions inside and outside the vehicle. Input data related to situations inside and outside the vehicle include sensing data from sensors, vehicle data obtained from the vehicle itself, user data obtained from vehicle users (wearable devices, etc.), data related to objects located inside and outside the vehicle, and data between the vehicle and the object. Event data related to vehicle-to-object situations such as collisions, environmental data related to the environment inside and outside the vehicle (location around the vehicle, weather, geography, road surface conditions, ambient brightness, traffic volume, etc.), other vehicles communicating with the vehicle/ It includes communication data processed or acquired by a server/base station/pedestrian terminal/infrastructure terminal, processed data processed by the data, and a pre-stored database.

In addition, as shown in FIG. 1, the processing units O1 and O2 generate object data included in the input data based on the plurality of object recognition models generated by the model generating unit P. Here, the model generator (P) performs deep learning on the input data received through the input unit (I), the storage unit (M), or the communication unit (T) to use the deep learning algorithm, and creates a learning model that has been trained. Enables the processor to process input data based on the corresponding learning model.

The learning model is an object recognition model, which is an object classification/identification/detection/recognition/object unit segmentation/semantic unit segmentation as an inference of deep learning (object classification, localization, detection, recognition, instance segmentation, semantic in that order). It is a model that infers the answer to the input and creates object data in order to perform object recognition such as segmentation).

Here, among the processing units, the first processing unit O1 may be a processing unit that processes input data based on a learning model generated by deep learning, and the second processing unit O2 is a non-deep learning model, that is, a non-deep learning model. It may be a processing unit that processes input data based on . However, here, the non-deep learning model of the second processing unit O2 includes a learning-based processing model and a non-learning-based processing model.

And as shown in Figure 1, the anomaly detection unit (A) extracts the object data for the object identified as the same among the object data, and compares the extracted object data to calculate an anomaly value. That is, it is determined whether result values of deep learning inference for the same object among the object data of the object included in the input data are similar to detect abnormality of the object recognition device. The premise here is that the object recognition reliability of the processor must be verified by comparing object data on the same object with the same object. In the end, the anomaly detection unit A determines that the corresponding processing unit has an anomaly when the object is identified but shows a result value that is far from zero.

Existing methods for diagnosing failures/abnormalities of processors use one type of artificial and fixed input data and compare the output data to verify the processor. However, the object recognition anomaly detection system of a vehicle according to an embodiment of the present invention compares object data for the same object in each processing unit based on input data related to the situation inside and outside the vehicle, unlike the existing diagnosis method. Detect abnormalities in object recognition. Therefore, the object recognition anomaly detection system of a vehicle according to an embodiment of the present invention compares the object recognition result of the second processing unit, which has a low operation amount and a high operation speed, to determine whether there is an abnormality in the first processing unit, which has a large operation amount and a slow operation speed. can be quickly detected. In addition, the object recognition anomaly detection system of a vehicle according to an embodiment of the present invention can accurately verify whether or not there is an anomaly by processing cases of various input data.

As used herein, "deep learning" is a type of machine learning and can be used interchangeably with machine learning. Machine learning is a technology related to systems and algorithms that improve their own performance while performing learning and reasoning based on empirical data. In other words, machine learning is a method of building a specific model to infer output data based on input data. Here, deep learning is a method of building such a specific model, and extracts features from input data between input data and output data through learning parameters such as weights, biases, and activation functions in algorithms such as artificial neural networks and transfers them to output data. It contains hidden data that Deep learning here includes supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

2 is a configuration diagram of a vehicle object recognition anomaly detection system according to an embodiment of the present invention, and the first processing unit O1 processes image data outside the vehicle such as Radar, LiDar, SVM-CAMERA, and CAMERA as input data can do. In particular, the first processor O1 extracts features of multiple objects included in input data and events between vehicles based on the deep learning model for object recognition generated by the model generator P. Through this, the first processing unit O1 can process and update data necessary for monitoring and controlling the vehicle so that the vehicle can drive itself without driver's or passenger's manipulation. Therefore, whether or not to diagnose the failure of the first processing unit O1 is very important since it determines the safety of the autonomous driving system.

And Figure 3 is a diagram showing the mechanism of the object recognition anomaly detection system of the vehicle according to an embodiment of the present invention. As shown in FIG. 3, the control unit C controls the first processing unit O1 of the selected processing unit to generate object data about an object among input data about situations inside and outside the vehicle with a deep learning-based object recognition algorithm. . Similarly, the control unit C controls the second processing unit O2 of the selected processing unit to generate object data about an object among input data about situations inside and outside the vehicle with a non-deep learning-based object recognition algorithm.

Then, the control unit C extracts an identically identified object from among each object data, and compares the object data of the corresponding object with each other. The control unit C determines whether or not the first processing unit O1 has an abnormality based on the result of the comparison. The control unit (C) can control another processor to replace the corresponding function when an abnormality in the first processing unit (O1) is detected, and the user interface (U) outputs this to the user and is connected to the server so that the user can Receive post-repair or inspection at a service center connected to the server.

Here, "~ part" in the embodiment of the present invention may be composed of each or a combination of software and hardware. In addition, "~ part" in this embodiment may be included in a computer-readable storage medium. In addition, part of "~ part" in this embodiment may be distributed and distributed to a plurality of hardware or software, or a combination thereof. In this case, the data processing of "-part" in this embodiment may be distributed, centralized, or accelerated, such as cloud computing or edge computing or AI edge computing.

In addition, “~ unit” used in this embodiment refers to software such as tasks, classes, subroutines, processes, objects, execution threads, programs, and program-based AI accelerators performed in a predetermined area on memory. individually or in combination, graphic processing units (GPUs), central processing units (CPUs), field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs), digital signal processors (DSPs), vision processing units (VPCs), neuromorphic It can be implemented individually or in combination with hardware such as chips (Neromorphic ICs) and hardware-based AI accelerators.

4 is a flow chart of a method for detecting an object recognition abnormality of a vehicle according to an embodiment of the present invention, comprising the steps of receiving input data related to situations inside and outside the vehicle from an input unit (S100 and S200); generating object data that is data about objects included in the input data based on the input data and a plurality of object recognition models in a processing unit (S100 and S200); and extracting object data for objects identified as the same among the object data by the anomaly detection unit (S300), and calculating an anomaly value by comparing the extracted object data (S400).

4 is a flow chart of a method for detecting an abnormality in object recognition of a vehicle according to an embodiment of the present invention, and in the step of receiving an input of the method for detecting anomaly in vehicle object recognition according to an embodiment of the present invention (S100, S200), the input data is It may be image data including objects inside and outside the vehicle. In the receiving step (S100, S200) of the object recognition abnormality detection method of the vehicle according to an embodiment of the present invention, the input data is the sensing data of the sensor measured inside or outside the vehicle or autonomous driving of the vehicle is performed with respect to the sensing data It may be processed data that is processed by doing.

And, before the step of receiving an input in the method for detecting an object recognition abnormality of a vehicle according to an embodiment of the present invention, a storage unit further includes a step (not shown) of storing sensing data of a sensor measured inside or outside the vehicle, , Object data, which is data about objects included in the input data and sensing data, may be generated based on the input data, sensing data, and a plurality of object recognition models in the generating step. Therefore, image data including objects inside and outside the vehicle may be used as pre-stored sensing data or real-time input data to diagnose a failure of the processing unit.

In addition, before the step of generating in the object recognition abnormality detection method of a vehicle according to an embodiment of the present invention, the control unit generates object data based on an object recognition model based on an algorithm other than deep learning among a plurality of processing units. A step of selecting two processing units may be further included, and in the step of selecting the second processing unit, the controller may select the processing unit having the lowest usage rate as the second processing unit. That is, among the processors, the second processing unit that has the lowest usage rate and processes input data based on a model other than deep learning can be selected, thereby minimizing resource release due to anomaly detection.

5 is a flowchart illustrating an anomaly detection mechanism of a method for detecting an anomaly in object recognition of a vehicle according to an embodiment of the present invention, and calculating an anomaly value in the method for detecting anomaly in vehicle object recognition according to an embodiment of the present invention ( In S400, an ideal value may be calculated as a difference between object data belonging to the same class among a plurality of object data (S300) extracted from the control unit (S400). That is, as a method of comparing the output data of each processing unit, it is possible to easily determine whether or not there is an abnormality as a result of four arithmetic operations between output data belonging to the same class. If it is judged as (-) during the four arithmetic operations and the subtraction of the output data is a value other than 0, and an ideal value whose absolute value is greater than or equal to the first reference value is calculated, the control unit infers information about the object included in the input data of the situation inside and outside the vehicle. It is possible to control the first processing unit to stop generating inference data (S500).

5 is a flowchart illustrating an anomaly detection mechanism of a method for detecting anomaly in vehicle object recognition according to an embodiment of the present invention, and in the controlling step (S500), an anomaly value (S402) continuously calculated over time When all of the abnormal values during the time or by the first number of times are equal to or greater than the first reference value, the first processing unit may be controlled to stop generating inference data about the object. In other words, by detecting the presence or absence of abnormalities by setting a limited time (number of times), it can be designed not to want to diagnose abnormalities too frequently according to the system design direction.

Detailed technical characteristics of each component of the vehicle object recognition anomaly detection system according to an embodiment of the present invention are the same as those of each step in the vehicle object recognition anomaly detection method according to an embodiment of the present invention described above. Or, since it is similar, a detailed description thereof will be omitted. Also, in the present invention, singular expressions include plural expressions unless the context clearly indicates otherwise.

In addition, ~ value here may be a point, a vector, or a matrix through it, and may be continuous or discrete according to a measurement or storage method.

Also, terms including ordinal numbers such as the first processing unit and the second processing unit may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

Although shown and described in relation to specific embodiments of the present invention as described above, it is understood that the present invention can be variously improved and changed without departing from the technical spirit of the present invention provided by the claims below. It will be apparent to those skilled in the art.

A: anomaly detection unit
C: control part
I: input
S: sensor
T: Ministry of Communications
P: model generator
U: user interface
O1: first processing unit
O2: second processing unit

Claims (15)

Receiving input data about conditions inside and outside the vehicle from an input unit;
generating object data that is data about an object included in the input data based on the input data and a plurality of object recognition models in a processing unit; and
A method for detecting an anomaly in object recognition of a vehicle, comprising: extracting object data about an object identified as the same among object data in an anomaly detection unit, and calculating an anomaly value by comparing the extracted object data. The method of claim 1,
at the stage of calculating
An object recognition anomaly detection method of a vehicle, characterized in that an anomaly value is calculated as a difference between object data belonging to the same class among a plurality of object data extracted from a control unit. The method of claim 2,
After the calculating step, if the abnormal value is greater than or equal to the first reference value, the control unit controls the processing unit to stop generating inference data about the object. The method of claim 3,
Object recognition anomaly detection of a vehicle, characterized in that in the controlling step, if the anomaly value continuously calculated over time is equal to or greater than the first reference value for a first time period, the first processing unit controls to stop generating inference data about the object. Way. The method of claim 1,
A method for detecting anomaly in object recognition of a vehicle, characterized in that in the receiving step, the input data is image data including objects inside and outside the vehicle. The method of claim 1,
In the receiving step, the input data is sensing data of a sensor measured inside or outside the vehicle or processing data processed by performing autonomous driving of the vehicle with respect to the sensing data. The method of claim 1,
Further comprising the step of storing the sensing data of the sensor measured inside or outside the vehicle in the storage unit before the step of receiving the input,
A method for detecting anomaly in object recognition in a vehicle, characterized in that in the generating step, object data that is data about objects included in the input data and sensing data is generated based on the input data, the sensing data, and a plurality of object recognition models. The method of claim 1,
In the generating step, a plurality of object recognition models include an object recognition model based on deep learning. The method of claim 8,
In the generating step, a first processing unit among a plurality of processing units generates object data based on an object recognition model based on deep learning, and a second processing unit generates object data based on an object recognition model based on an algorithm other than deep learning. An object recognition anomaly detection method of a vehicle, characterized in that for generating data. The method of claim 9,
Prior to the generating step, the control unit further comprises selecting a second processing unit to generate object data based on an object recognition model based on an algorithm other than deep learning among a plurality of processing units,
In the step of selecting the first processing unit, the control unit selects the processing unit with the lowest usage rate as the second processing unit. an input unit that receives input data about conditions inside and outside the vehicle;
a processing unit that generates object data, which is data about an object included in the input data, based on the input data and a plurality of object recognition models; and
An anomaly detection unit for extracting object data for an object identified as the same among object data and comparing the extracted object data to calculate an anomaly value; The method of claim 11,
The object recognition anomaly detection system of a vehicle, characterized in that the anomaly detection unit calculates an anomaly value as a difference between object data belonging to the same class among a plurality of extracted object data. The method of claim 12,
The object recognition anomaly detection system of a vehicle, further comprising a control unit controlling the processing unit to stop generating inference data about an object when the abnormal value is equal to or greater than the first reference value. The method of claim 11,
The plurality of object recognition models include an object recognition model based on deep learning,
Among the plurality of processing units, the first processing unit generates object data based on an object recognition model based on deep learning, and the second processing unit generates object data based on an object recognition model based on an algorithm other than deep learning. Object recognition anomaly detection system of a vehicle, characterized in that. The method of claim 13,
Further comprising a control unit for processing to select a second processing unit to generate object data based on an object recognition model based on an algorithm other than deep learning among the plurality of processing units;
The object recognition anomaly detection system of a vehicle, characterized in that the control unit processes to select the processing unit with the lowest usage rate as the second processing unit.

Images