CN116012381A

CN116012381A - Method and system for alarming foreign matters falling from vehicle

Info

Publication number: CN116012381A
Application number: CN202310303165.5A
Authority: CN
Inventors: 黄兵; 袁勇; 白皓; 张光锦; 罗世豪; 靳庆浩; 刘勇; 廖知勇; 蔡艾宏; 唐浩
Original assignee: Sichuan Expressway Construction And Development Group Co ltd
Current assignee: Sichuan Expressway Construction And Development Group Co ltd
Priority date: 2023-03-27
Filing date: 2023-03-27
Publication date: 2023-04-25
Anticipated expiration: 2043-03-27
Also published as: CN116012381B

Abstract

The invention provides a method and a system for alarming foreign matters falling from a vehicle, and relates to the technical field of artificial intelligence. In the invention, the target physical distance information of the target foreign matter is obtained; acquiring to-be-processed foreign matter image data detected by the image detection device based on the foreign matter detection information under the condition that the target physical distance information is smaller than the reference physical distance information; digging out a foreign object image characteristic representation of the foreign object image data to be processed by using a foreign object image analysis network; analyzing corresponding foreign object image classification data based on the foreign object image feature representation and the reference foreign object image feature representation, wherein the reference foreign object image feature representation is formed by mining the foreign object image data of the reference foreign object through a foreign object image analysis network, and the foreign object image classification data is used for reflecting foreign object type information of the target foreign object; and performing a foreign object alarm operation based on the foreign object image classification data. Based on the above, the reliability of the foreign matter alarm can be improved to some extent.

Description

Method and system for alarming foreign matters falling from vehicle

Technical Field

The invention relates to the technical field of artificial intelligence, in particular to a method and a system for alarming foreign matters falling from a vehicle.

Background

Artificial intelligence is a branch of the computer science, and has been known as one of the three-most advanced technologies in the world (space technology, energy technology, artificial intelligence) since the seventies of the twentieth century, and has also been known as one of the three-most advanced technologies in the twenty-first century (genetic engineering, nanoscience, artificial intelligence). This is because it has been rapidly developed over the last three decades, has been widely used in many disciplines and has achieved great success, and artificial intelligence has evolved into a single branch, both theoretically and practically self-contained. Artificial intelligence is a discipline of studying certain mental processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) that make a computer simulate a person, and mainly includes the principle of computer-implemented intelligence, manufacturing a computer similar to human brain intelligence, so that the computer can implement higher-level application. Artificial intelligence will involve computer science, psychology, philosophy, and linguistics. It can be said that almost all subjects of natural science and social science are far beyond the category of computer science, the relationship between artificial intelligence and thinking science is the relationship between practice and theory, the artificial intelligence is in the technical application level of thinking science, and the artificial intelligence is an application branch of the artificial intelligence. Mathematics are often considered as the basic science of a variety of disciplines, which also enter the language and thinking fields, and artificial intelligence disciplines must also borrow mathematical tools, which not only function in the context of standard logic, fuzzy mathematics, etc., but also enter artificial intelligence disciplines, which will promote each other and develop more rapidly.

In many applications of artificial intelligence, including analysis of images based on artificial intelligence, such as foreign matter recognition, there is a problem in the prior art that reliability of foreign matter alarm is poor in a process of performing a vehicle drop foreign matter alarm based on image analysis.

Disclosure of Invention

Accordingly, the present invention is directed to a method and a system for alarming a vehicle for a falling foreign object, so as to improve the reliability of the foreign object alarm to a certain extent.

In order to achieve the above purpose, the embodiment of the present invention adopts the following technical scheme:

the method for alarming the foreign matters falling from the vehicle comprises the following steps:

transmitting foreign matter detection information to radar detection equipment, and after receiving foreign matter existence information reported by the radar detection equipment based on the foreign matter detection information under the condition that the foreign matter is detected, analyzing the foreign matter existence information to obtain target physical distance information of target foreign matters, wherein the target physical distance information is used for reflecting the distance between a target vehicle and the target foreign matters;

issuing foreign matter detection information to an image detection device and acquiring to-be-processed foreign matter image data detected by the image detection device based on the foreign matter detection information under the condition that the target physical distance information is smaller than reference physical distance information, wherein the to-be-processed foreign matter image data comprises at least one frame of to-be-processed foreign matter image of the target foreign matter;

Digging out foreign object image characteristic representations of the foreign object image data to be processed by using a foreign object image analysis network, wherein the foreign object image analysis network is formed by performing corresponding network optimization operation based on a first network learning cost index and a second network learning cost index, the first network learning cost index is a classified learning cost index, the second network learning cost index is other learning cost indexes except the classified learning cost index, and the first network learning cost index and the second network learning cost index are formed by performing corresponding analysis according to deep image characteristic representations of each foreign object image data;

analyzing corresponding foreign object image classification data based on the foreign object image feature representation and a reference foreign object image feature representation, wherein the reference foreign object image feature representation is formed by mining foreign object image data of reference foreign objects through the foreign object image analysis network, and the foreign object image classification data is used for reflecting foreign object type information of the target foreign objects;

and performing foreign matter alarm operation based on the foreign matter image classification data.

In some preferred embodiments, in the above vehicle drop foreign object warning method, the step of analyzing the corresponding foreign object image classification data based on the foreign object image feature representation and the reference foreign object image feature representation includes:

Analyzing corresponding feature representation distances based on the foreign object image feature representations and the reference foreign object image feature representations;

marking the reference foreign object type information of the reference foreign object as the foreign object type information of the target foreign object corresponding to the foreign object image data to be processed under the condition that the characteristic representing distance is smaller than or equal to a predetermined characteristic representing first reference distance so as to form corresponding foreign object image classification data;

and marking the reference foreign matter type information of the foreign matters except the reference foreign matters as the foreign matter type information of the target foreign matters corresponding to the to-be-processed foreign matter image data under the condition that the characteristic representation distance is larger than the characteristic representation first reference distance so as to form corresponding foreign matter image classification data.

In some preferred embodiments, in the above-described vehicle falling foreign matter alarm method, the vehicle falling foreign matter alarm method further includes:

extracting a typical foreign matter image data cluster, wherein the typical foreign matter image data cluster comprises at least one foreign matter image data corresponding to a typical foreign matter;

analyzing a corresponding target network learning cost index based on foreign object image data corresponding to each typical foreign object in the typical foreign object image data cluster, wherein the target network learning cost index comprises a first network learning cost index and a second network learning cost index, the first network learning cost index is a classification learning cost index, the second network learning cost index is other learning cost indexes except the classification learning cost index, the first network learning cost index and the second network learning cost index are formed by corresponding analysis according to deep image feature representation of each foreign object image data, the deep image feature representation of each foreign object image data is formed by mining based on a convolutional network model with gradient optimization operation, a first feature mining unit in the convolutional network model with gradient optimization operation comprises a special feature mining operator, and the target dimension comprises a time dimension;

And optimizing based on the target network learning cost index to form a foreign object image analysis network.

In some preferred embodiments, in the vehicle drop foreign object warning method, the step of analyzing the corresponding target network learning cost indicator based on the foreign object image data corresponding to each of the typical foreign objects in the typical foreign object image data cluster includes:

respectively mining deep image characteristic representations corresponding to each piece of foreign object image data based on the foreign object image data corresponding to each piece of typical foreign object in the typical foreign object image data cluster by using the convolution network model with gradient optimization operation;

determining corresponding linear dimension importance parameter distribution based on the foreign object image data corresponding to each typical foreign object in the typical foreign object image data cluster;

and analyzing a corresponding first index of the network learning cost based on the deep image characteristic representation corresponding to each piece of foreign object image data and the linear dimension importance parameter distribution.

In some preferred embodiments, in the vehicle drop foreign object warning method, the step of analyzing the corresponding first index of the network learning cost based on the deep image feature representation and the linear dimension importance parameter distribution corresponding to each of the foreign object image data includes:

For each piece of foreign object image data, determining distribution ordering information of typical foreign objects corresponding to the foreign object image data, determining column parameter distribution corresponding to the distribution ordering information from the linear dimension importance parameter distribution, performing transposition operation on the column parameter distribution to obtain corresponding transposition parameter distribution, performing multiplication operation on the transposition parameter distribution and deep image characteristic representation corresponding to the foreign object image data to output corresponding multiplication operation results, performing superposition operation on the multiplication operation results and shift parameters configured in advance for the distribution ordering information to output corresponding superposition operation results, performing exponential operation on the superposition operation results to output exponential operation results corresponding to the foreign object image data, and performing column parameter distribution in the linear dimension importance parameter distribution to correspond to each typical foreign object in the typical foreign object image data cluster;

performing superposition operation on the index operation result corresponding to each piece of foreign object image data to output a corresponding target index operation result, and performing ratio calculation on the index operation result corresponding to each piece of foreign object image data and the target index operation result respectively to output a ratio calculation result corresponding to each piece of foreign object image data;

And performing superposition operation on the ratio calculation result corresponding to each piece of foreign object image data to output a corresponding superposition operation result, and determining a first index of the network learning cost based on the superposition operation result, wherein the first index of the network learning cost is inversely related to the superposition operation result.

analyzing corresponding feature representation change parameters based on deep image feature representations corresponding to each piece of foreign object image data;

analyzing corresponding intermediate image data representing parameters based on the characteristic representing change parameters and the initial image data representing parameters;

and analyzing a second index of the network learning cost based on the deep image characteristic representation corresponding to each piece of foreign object image data and the intermediate image data representing parameters.

In some preferred embodiments, in the vehicle drop foreign object alarm method, the step of using the convolutional network model with gradient optimization operation to respectively mine out deep image feature representations corresponding to each of the foreign object image data based on the foreign object image data corresponding to each of the typical foreign object image data clusters includes:

performing feature mining operation on foreign object image data corresponding to typical foreign objects in the typical foreign object image data cluster by using a plurality of feature mining units included in the convolution network model with gradient optimization operation so as to obtain a plurality of initial image feature representations corresponding to the foreign object image data;

clustering a plurality of initial image feature representations corresponding to the foreign object image data to form at least one corresponding clustering center feature representation to be processed;

performing gradient optimization operation on a plurality of initial image feature representations corresponding to the foreign object image data according to the at least one clustering center feature representation to be processed so as to form a deep image feature representation corresponding to the foreign object image data;

the step of performing gradient optimization operation on the plurality of initial image feature representations corresponding to the foreign object image data according to the at least one clustering center feature representation to be processed to form a deep image feature representation corresponding to the foreign object image data includes:

For each cluster-to-be-processed central feature representation: analyzing a plurality of initial image feature representations corresponding to the foreign object image data by using a feature optimization unit included in the convolution network model with gradient optimization operation, wherein the initial optimization parameters are included in the cluster center feature representation to be processed, and performing parameter mapping operation on the initial optimization parameters corresponding to the cluster center feature representation to be processed by using a parameter mapping unit included in the convolution network model with gradient optimization operation, so as to form parameter mapping data corresponding to the cluster center feature representation to be processed;

and performing global parameter mapping operation on the parameter mapping data corresponding to each to-be-processed clustering center feature representation by using the parameter mapping unit so as to form a deep image feature representation corresponding to the foreign object image data.

In some preferred embodiments, in the method for warning of foreign object falling in a vehicle, the step of analyzing the corresponding feature representation change parameter based on the deep image feature representation corresponding to each of the foreign object image data includes:

for each piece of foreign object image data, carrying out average value calculation on deep image characteristic representations corresponding to all pieces of foreign object image data so as to output corresponding initial image data representing parameters, and carrying out difference calculation on the initial image data representing parameters and the deep image characteristic representations corresponding to the foreign object image data so as to output difference calculation results corresponding to the foreign object image data;

Respectively carrying out excitation mapping output on the difference calculation result corresponding to each piece of foreign object image data so as to obtain an excitation mapping result corresponding to each piece of foreign object image data;

performing superposition operation on excitation mapping results corresponding to each piece of foreign object image data to output corresponding superposition excitation mapping results, and performing superposition operation on the superposition excitation mapping results and preset standard parameters to output corresponding target superposition operation results;

calculating the sum value between the superposition excitation mapping result and the target superposition operation result to output corresponding characteristic representation change parameters;

the step of analyzing the corresponding intermediate image data representing parameters based on the characteristic representing variation parameters and the initial image data representing parameters comprises the steps of:

extracting a predetermined optimization progress control hyper-parameter;

determining a negative correlation progress control superparameter corresponding to the optimization progress control superparameter, wherein the sum value between the negative correlation progress control superparameter and the optimization progress control superparameter is equal to 1;

performing superposition operation on the negative correlation progress control superparameter and the characteristic representation change parameter to output a corresponding superposition operation result, and performing superposition operation on the superposition operation result and the initial image data representation parameter to output a corresponding intermediate image data representation parameter;

The step of analyzing the second index of the network learning cost based on the deep image characteristic representation corresponding to each piece of foreign object image data and the intermediate image data representing parameters comprises the following steps:

for each deep image characteristic representation corresponding to the foreign object image data, performing a difference calculation on the deep image characteristic representation and the intermediate image data representative parameters to output a difference image characteristic representation corresponding to the deep image characteristic representation;

for each deep image characteristic representation corresponding to the foreign object image data, performing square summation calculation on each characteristic representation parameter in the difference image characteristic representation corresponding to the deep image characteristic representation to output a square summation calculation result corresponding to the deep image characteristic representation;

and performing sum calculation on the square summation calculation result corresponding to the deep image characteristic representation corresponding to each piece of foreign object image data so as to output a corresponding second index of the network learning cost.

Analyzing a first index of corresponding network learning cost based on the foreign object image data corresponding to each typical foreign object in the typical foreign object image data cluster;

analyzing a second index of the corresponding network learning cost based on the foreign object image data corresponding to each typical foreign object in the typical foreign object image data cluster;

and carrying out weighted summation operation on the first index of the network learning cost and the second index of the network learning cost to output a corresponding target network learning cost index, wherein the weighting coefficient corresponding to the first index of the network learning cost is larger than that corresponding to the second index of the network learning cost.

The embodiment of the invention also provides a vehicle falling foreign matter alarm system, which comprises a processor and a memory, wherein the memory is used for storing a computer program, and the processor is used for executing the computer program so as to realize the vehicle falling foreign matter alarm method.

The method and the system for alarming the falling foreign matters of the vehicle can obtain the target physical distance information of the target foreign matters; acquiring to-be-processed foreign matter image data detected by the image detection device based on the foreign matter detection information under the condition that the target physical distance information is smaller than the reference physical distance information; digging out a foreign object image characteristic representation of the foreign object image data to be processed by using a foreign object image analysis network; analyzing corresponding foreign object image classification data based on the foreign object image feature representation and the reference foreign object image feature representation; and performing a foreign object alarm operation based on the foreign object image classification data. Based on the foregoing, since the foreign object image analysis network performs the corresponding network optimization operation based on the first index of the network learning cost and the second index of the network learning cost to form the foreign object image analysis network, the first index of the network learning cost is the classified learning cost index, and the second index of the network learning cost is other learning cost indexes except the classified learning cost index, i.e. the network optimization operation is performed from two different dimensions, the accuracy of the formed foreign object image analysis network can be relatively high, the reliability of the foreign object image feature representation of the excavated foreign object image data to be processed can be ensured, the reliability of the foreign object image classified data analysis can be improved, the reliability of the foreign object alarm operation can be further ensured, and the defects of the prior art can be improved.

In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments accompanied with figures are described in detail below.

Drawings

Fig. 1 is a block diagram of a system for alarming a foreign object falling from a vehicle according to an embodiment of the present invention.

Fig. 2 is a flowchart illustrating steps involved in a method for warning a vehicle of a foreign object falling in an embodiment of the present invention.

Fig. 3 is a schematic diagram of each module included in the vehicle falling foreign matter alarm device according to the embodiment of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.

Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

As shown in fig. 1, an embodiment of the present invention provides a vehicle drop foreign matter alarm system. Wherein, the vehicle drop foreign matter alarm system may include a memory and a processor.

In detail, the memory and the processor are electrically connected directly or indirectly to realize transmission or interaction of data. For example, electrical connection may be made to each other via one or more communication buses or signal lines. The memory may store at least one software functional module (computer program) that may exist in the form of software or firmware. The processor may be configured to execute an executable computer program stored in the memory, so as to implement the method for warning of foreign objects falling from a vehicle according to the embodiment of the present invention.

It is to be appreciated that in some embodiments, the Memory may be, but is not limited to, random access Memory (Random Access Memory, RAM), read Only Memory (ROM), programmable Read Only Memory (Programmable Read-Only Memory, PROM), erasable Read Only Memory (ErasableProgrammable Read-Only Memory, EPROM), electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), and the like. The processor may be a general purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (NetworkProcessor, NP), a System on Chip (SoC), etc.; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

It will be appreciated that in some embodiments, the vehicle drop foreign object alarm system may be a server with data processing capabilities.

Referring to fig. 2, an embodiment of the present invention further provides a method for alarming a foreign object falling from a vehicle, which can be applied to the system for alarming a foreign object falling from a vehicle. The method steps defined by the flow related to the vehicle foreign object falling alarm method can be realized by the vehicle foreign object falling alarm system.

The specific flow shown in fig. 2 will be described in detail.

Step S110, transmitting foreign matter detection information to the radar detection equipment, and after receiving the foreign matter existence information reported by the radar detection equipment based on the foreign matter detection information under the condition that the foreign matter is detected, analyzing the foreign matter existence information to obtain target physical distance information of the target foreign matter.

In the embodiment of the invention, the vehicle falling foreign matter alarm system can send out foreign matter detection information to the radar detection equipment, and after receiving the foreign matter existence information reported by the radar detection equipment based on the foreign matter detection information under the condition of detecting the foreign matter, the foreign matter existence information is analyzed to obtain the target physical distance information of the target foreign matter. The target physical distance information is used to reflect a distance between a target vehicle and the target foreign matter. The radar detection device may be a communicatively connected millimeter wave radar device, for example.

And step S120, issuing foreign matter detection information to an image detection device and acquiring to-be-processed foreign matter image data detected by the image detection device based on the foreign matter detection information under the condition that the target physical distance information is smaller than the reference physical distance information.

In the embodiment of the invention, the vehicle falling foreign matter alarm system can issue foreign matter detection information to the image detection device and acquire to-be-processed foreign matter image data detected by the image detection device based on the foreign matter detection information under the condition that the target physical distance information is smaller than the reference physical distance information. The foreign matter image data to be processed includes at least one frame of foreign matter image to be processed of the target foreign matter. The specific value of the reference physical distance information can be configured according to actual requirements.

And step S130, utilizing a foreign object image analysis network to mine out the foreign object image characteristic representation of the foreign object image data to be processed.

In the embodiment of the invention, the vehicle falling foreign matter alarm system can utilize a foreign matter image analysis network to dig out the foreign matter image characteristic representation of the foreign matter image data to be processed. The foreign object image analysis network performs corresponding network optimization operation based on a first index of network learning cost and a second index of network learning cost to form the foreign object image analysis network, wherein the first index of network learning cost is a classified learning cost index, the second index of network learning cost is other learning cost indexes except the classified learning cost index, and the first index of network learning cost and the second index of network learning cost perform corresponding analysis according to deep image characteristic representation of each foreign object image data to form the foreign object image analysis network.

Step S140, analyzing corresponding foreign object image classification data based on the foreign object image feature representation and the reference foreign object image feature representation.

In the embodiment of the invention, the vehicle falling foreign matter alarm system can analyze corresponding foreign matter image classification data based on the foreign matter image feature representation and the reference foreign matter image feature representation. The reference foreign object image feature represents that the foreign object image data of the reference foreign object is mined through the foreign object image analysis network to form the foreign object image classification data, wherein the foreign object image classification data is used for reflecting foreign object type information of the target foreign object, such as description of the target foreign object, so that more information of the target foreign object, such as names and attributes, such as dangerous attributes, of the target foreign object can be acquired.

And step S150, performing foreign matter alarm operation based on the foreign matter image classification data.

In the embodiment of the invention, the vehicle falling foreign matter alarm system can perform foreign matter alarm operation based on the foreign matter image classification data, for example, when in alarm, the foreign matter image classification data and the to-be-processed foreign matter image data can be generally sent to a terminal device for display, and the position of the target foreign matter can be sent to the terminal device for display so as to give an alarm to a driver corresponding to the terminal device for reminding avoidance, and meanwhile, the vehicle falling foreign matter alarm system can also be used for reminding a road patrol personnel to timely check and process in real time, so that the road patrol investment can be reduced.

Based on the foregoing (i.e., the foregoing steps S110 to S150), since the foreign object image analysis network performs the corresponding network optimization operation based on the first index of the network learning cost and the second index of the network learning cost to form the foreign object image analysis network, the first index of the network learning cost is the classified learning cost index, and the second index of the network learning cost is other learning cost indexes than the classified learning cost index, i.e., the network optimization operation is performed from two different dimensions, the accuracy of the formed foreign object image analysis network can be relatively high, the reliability of the foreign object image feature representation of the mined foreign object image data to be processed can be ensured, the reliability of the foreign object image classification data analysis can be improved, the reliability of the foreign object alarm operation can be further ensured, and the defects of the prior art can be improved.

In addition, the technology such as video and radar is utilized for identification, and accurate positioning analysis can be performed on the dropped object, for example, the dropped object is positioned to the level of a lane, so that foreign matter avoidance early warning can be performed through navigation software or an electronic display screen on a road; and after informing the road patrol personnel to timely clean the road, namely after cleaning the foreign matters, the early warning is automatically canceled (for example, cleaning actions and cleaning effects can be automatically identified), and cleaning time is recorded, so that the road patrol personnel working efficiency and quality can be checked.

It will be appreciated that in some embodiments, the step S140 may further include the following specific sub-steps:

analyzing corresponding feature representation distances, such as cosine distances between feature representations, based on the foreign object image feature representations and the reference foreign object image feature representations;

marking the reference foreign matter type information of the reference foreign matter as the foreign matter type information of the target foreign matter corresponding to the foreign matter image data to be processed to form corresponding foreign matter image classification data under the condition that the characteristic representing distance is smaller than or equal to a predetermined characteristic representing first reference distance, wherein the specific value of the characteristic representing the first reference distance can be configured according to actual requirements;

in the case where the feature representing distance is larger than the feature representing first reference distance, the reference foreign matter type information of the foreign matter other than the reference foreign matter is marked as the foreign matter type information of the target foreign matter corresponding to the foreign matter image data to be processed to form the corresponding foreign matter image classification data, the reference foreign matter may be configured with various types, the reference foreign matter type information of the one reference foreign matter whose corresponding feature representing distance is smallest may be marked as the foreign matter type information of the target foreign matter corresponding to the foreign matter image data to be processed, or in the case where all the feature representing distances are larger than the feature representing first reference distance, the foreign matter type information of the target foreign matter corresponding to the foreign matter image data to be processed may be determined as the unidentified type information.

It can be appreciated that, in some embodiments, in order to make the feature mining capability of the foreign object image analysis network higher and ensure the mining precision thereof, the vehicle foreign object falling alarm method may further include the following specific steps:

extracting a typical foreign object image data cluster, wherein the typical foreign object image data cluster comprises at least one foreign object image data corresponding to a typical foreign object, one type of foreign object information can correspond to at least one typical foreign object, and one typical foreign object corresponds to at least one foreign object image data;

analyzing a corresponding target network learning cost index based on foreign object image data corresponding to each typical foreign object in the typical foreign object image data cluster, wherein the target network learning cost index comprises a first network learning cost index and a second network learning cost index, the first network learning cost index is a classified learning cost index, the second network learning cost index is other learning cost indexes except the classified learning cost index, the first network learning cost index and the second network learning cost index are formed by carrying out corresponding analysis according to deep image feature representation of each foreign object image data, the deep image feature representation of each foreign object image data is formed by carrying out mining based on a convolutional network model with gradient optimization operation, a first feature mining unit in the convolutional network model with gradient optimization operation comprises a special feature mining operator, the special feature mining operator is used for carrying out feature mining operation on data with a target dimension which is larger than that of data with other dimensions, the target dimension comprises a time dimension, the special feature mining operator can refer to special feature mining operator, and the special feature mining operator can refer to a non-symmetrical feature mining operator (Asymmetric Convolution can refer to m, n and n, n are different from m;

And optimizing based on the target network learning cost index to form a foreign object image analysis network, such as optimizing along the direction of reducing the target network learning cost index.

It will be appreciated that, in some embodiments, the step of analyzing the target network learning cost index based on the foreign object image data corresponding to each of the representative foreign objects in the representative foreign object image data cluster may further include the following specific sub-steps:

respectively mining deep image characteristic representations corresponding to each piece of foreign object image data based on the foreign object image data corresponding to each piece of typical foreign object in the typical foreign object image data cluster by using the convolution network model with gradient optimization operation, as described in the following related description;

determining a corresponding linear dimension importance parameter distribution based on the foreign object image data corresponding to each of the representative foreign object image data clusters, wherein a column parameter distribution in the linear dimension importance parameter distribution corresponds to each of the representative foreign objects in the representative foreign object image data clusters, for example, the column parameter distribution can be used for reflecting the similarity between the representative foreign object and the foreign object image data corresponding to each other representative foreign object;

It may be appreciated that, in some embodiments, the step of analyzing the corresponding first index of the network learning cost based on the deep image feature representation and the linear dimension importance parameter distribution corresponding to each of the foreign object image data may further include the following specific sub-steps:

for each piece of foreign object image data, determining distribution ordering information of typical foreign objects corresponding to the foreign object image data, such as a first column, a second column, a third column, a fourth column and the like, determining column parameter distribution corresponding to the distribution ordering information from the linear dimension importance parameter distribution, such as a first column, a second column, a third column, a fourth column and the like, performing transposition operation on the column parameter distribution, namely, the line-to-column exchange of parameters, to obtain corresponding transposition parameter distribution, performing multiplication operation on the transposition parameter distribution and deep image feature representation corresponding to the foreign object image data, to output corresponding multiplication operation results, performing superposition operation on the multiplication operation results and shift parameters configured for the distribution ordering information in advance, to output corresponding superposition operation results, performing exponential operation on the superposition operation results, to output exponential operation results corresponding to the foreign object image data, wherein the column parameter distribution in the linear dimension importance parameter distribution corresponds to each typical foreign object in the cluster of the foreign object image data, and the displacement parameter can be used as an optimized object;

and performing a superposition operation on the ratio calculation result corresponding to each piece of foreign object image data (or alternatively, performing a logarithm taking operation on the ratio calculation result corresponding to each piece of foreign object image data, and then performing a superposition operation on the logarithm taking result) so as to output a corresponding superposition operation result, and determining a first index of network learning cost based on the superposition operation result, where the first index of network learning cost is inversely related to the superposition operation result, for example, a product of the superposition operation result and negative one may be used as the first index of network learning cost.

It will be appreciated that, in some embodiments, the step of mining the deep image feature representation corresponding to each of the plurality of foreign object image data based on the foreign object image data corresponding to each of the plurality of foreign object image data clusters by using the convolutional network model with gradient optimization operation may further include the following specific sub-steps:

performing feature mining operation on foreign object image data corresponding to typical foreign objects in the typical foreign object image data cluster by using a plurality of feature mining units included in the convolutional network model with gradient optimization operation so as to obtain a plurality of initial image feature representations corresponding to the foreign object image data, wherein each feature mining unit corresponds to one initial image feature representation;

Clustering a plurality of initial image feature representations corresponding to the foreign object image data, wherein a specific clustering algorithm is not limited, so as to form at least one corresponding clustering center feature representation to be processed;

and performing gradient optimization operation on a plurality of initial image feature representations corresponding to the foreign object image data according to the at least one clustering center feature representation to be processed so as to form a deep image feature representation corresponding to the foreign object image data.

It will be appreciated that, in some embodiments, the step of performing a gradient optimization operation on the plurality of initial image feature representations corresponding to the foreign object image data according to the at least one cluster center feature representation to be processed to form a deep image feature representation corresponding to the foreign object image data may further include the following specific sub-steps:

for each cluster-to-be-processed central feature representation: analyzing a plurality of initial image feature representations corresponding to the foreign object image data by using a feature optimization unit included in the convolutional network model with gradient optimization operation, wherein the initial optimization parameters are included in the clustering center feature representation to be processed, and performing parameter mapping operation on the initial optimization parameters corresponding to the clustering center feature representation to be processed by using a parameter mapping unit included in the convolutional network model with gradient optimization operation to form parameter mapping data corresponding to the clustering center feature representation to be processed, namely, mapping each parameter in the initial optimization parameters to an interval, such as 0-1;

Therein, it will be appreciated that in some embodiments, the central feature representation for each cluster to be processed: the step of analyzing the initial optimization parameters of the plurality of initial image feature representations corresponding to the foreign object image data for the to-be-processed cluster center feature representation by using a feature optimization unit included in the convolutional network model with gradient optimization operation may further include the following specific sub-steps:

performing filtering operation on each initial image feature representation corresponding to the foreign object image data by using a feature optimization unit included in the convolution network model with gradient optimization operation, wherein the filtering operation can be realized by using a filtering matrix or a filtering operator included in the initial image feature representation to output a corresponding filtering image feature representation;

using the feature optimization unit to represent each initial image feature representation corresponding to the foreign object image data into a target estimation operation, such as based on a softmax function, to output corresponding target estimation data;

Analyzing a difference feature representation between each initial image feature representation corresponding to the foreign object image data and the cluster center feature representation to be processed by utilizing the feature optimization unit;

the target estimation data corresponding to each initial image feature representation of the foreign object image data is marked as an influence parameter, so that the difference feature representation corresponding to each initial image feature representation of the foreign object image data is fused, and initial optimization parameters of a plurality of initial image feature representations of the foreign object image data for the cluster center feature representation to be processed are output, that is, the difference feature representations can be weighted and overlapped based on the influence parameter.

It may be understood that, in some embodiments, the step of performing global parameter mapping operation on the parameter mapping data corresponding to each of the cluster center feature representations to be processed by using the parameter mapping unit to form a deep image feature representation corresponding to the foreign object image data may further include the following specific sub-steps:

performing global parameter mapping operation on the parameter mapping data corresponding to each to-be-processed cluster center feature representation by using the parameter mapping unit to output candidate deep image feature representations corresponding to the foreign object image data, that is, the candidate deep image feature representations corresponding to the foreign object image data are formed by global parameter mapping operation results of the parameter mapping data corresponding to each to-be-processed cluster center feature representation, wherein the global parameter mapping operation can be that a sum value of each parameter is calculated, and then, the ratio of the parameter to the sum value is used as a global parameter mapping operation result;

Performing a parameter decimation operation, such as a Pooling operation, which may be specifically the largest value, on the candidate deep image feature representation to form a decimated image feature representation of the foreign object image data corresponding to each feature Channel (Channel);

performing cascading operation and linear integration operation on the deep image feature representation of the foreign object image data corresponding to the decimated image feature representation of each feature channel to output linear integration parameters of the deep image feature representation of the foreign object image data corresponding to each feature channel, that is, performing cascading operation, that is, splicing, on the decimated image feature representation of each feature channel, and then performing linear integration on the cascading result, for example, through an MLP unit;

and multiplying the linear integration parameter corresponding to each characteristic channel of the deep image characteristic representation of the foreign object image data and the characteristic representation of each characteristic channel in the deep image characteristic representation of the foreign object image data to form the deep image characteristic representation corresponding to the foreign object image data.

It may be appreciated that, in some embodiments, the step of using the convolutional network model with gradient optimization operation to extract deep image feature representations corresponding to each of the typical foreign object image data based on the foreign object image data corresponding to each of the typical foreign object image data clusters, may further include the following specific sub-steps:

sequentially performing feature mining operation on the foreign object image data corresponding to the typical foreign object in the typical foreign object image data cluster by using a plurality of feature mining units included in the convolution network model with gradient optimization operation so as to obtain an initial image feature representation corresponding to the foreign object image data, wherein the feature mining units are connected in cascade, and the initial image feature representation corresponding to the foreign object image data is data output by the last feature mining unit;

performing feature representation compression operation on the initial image feature representation to form a corresponding compressed image feature representation, so that the dimension of the compressed image feature representation is smaller than that of the initial image feature representation, and compressing the dimension of the feature representation is realized;

Performing a convolution operation on the compressed image feature representation to form a corresponding convolved image feature representation, and performing a feature representation magnification operation on the convolved image feature representation to form a corresponding magnified image feature representation such that a dimension of the magnified image feature representation is greater than a dimension of the convolved image feature representation and equal to a dimension of the initial image feature representation;

and performing superposition operation on the amplified image characteristic representation and the initial image characteristic representation to form a deep image characteristic representation corresponding to the foreign object image data.

It will be appreciated that, in some embodiments, the step of analyzing the corresponding feature representation change parameter based on the deep image feature representation corresponding to each of the foreign object image data may further include the following specific sub-steps:

Respectively carrying out excitation mapping output, namely activation, on the difference calculation result corresponding to each piece of foreign object image data to obtain an excitation mapping result corresponding to each piece of foreign object image data;

performing superposition operation on the excitation mapping result corresponding to each piece of foreign object image data to output a corresponding superposition excitation mapping result, and performing superposition operation on the superposition excitation mapping result and a pre-configured standard parameter to output a corresponding target superposition operation result, wherein the standard parameter can be fixed, such as 1, and can also be used as a network optimization object;

and calculating the sum value between the superposition excitation mapping result and the target superposition operation result to output corresponding characteristic representation change parameters.

It will be appreciated that in some embodiments, the step of analyzing the corresponding intermediate image data representative parameter based on the feature representation change parameter and the initial image data representative parameter may further include the following specific sub-steps:

extracting a predetermined optimization progress control hyper-parameter, such as a learning rate;

And performing superposition operation on the negative correlation progress control superparameter and the characteristic representation change parameter to output a corresponding superposition operation result, and performing superposition operation on the superposition operation result and the initial image data representation parameter to output a corresponding intermediate image data representation parameter.

It will be appreciated that, in some embodiments, the step of analyzing the second index of the network learning cost based on the deep image feature representation corresponding to each of the foreign object image data and the intermediate image data representing parameters may further include the following specific sub-steps:

based on the foreign object image data corresponding to each typical foreign object in the typical foreign object image data cluster, analyzing a corresponding first index of network learning cost, as described in the previous description;

based on the foreign object image data corresponding to each typical foreign object in the typical foreign object image data cluster, analyzing a corresponding second index of the network learning cost, as described in the previous related description;

and performing weighted summation operation on the first index of the network learning cost and the second index of the network learning cost to output a corresponding target network learning cost index, wherein the weighted coefficient corresponding to the first index of the network learning cost is larger than the weighted coefficient corresponding to the second index of the network learning cost.

Referring to fig. 3, an embodiment of the invention further provides a device for alarming a foreign object falling from a vehicle, which can be applied to the system for alarming a foreign object falling from a vehicle. Wherein, the vehicle drop foreign matter alarm device may include:

the foreign matter information analysis module is used for issuing foreign matter detection information to the radar detection equipment, and after receiving the foreign matter existence information reported by the radar detection equipment based on the foreign matter detection information under the condition that the foreign matter is detected, analyzing the foreign matter existence information to obtain target physical distance information of target foreign matters, wherein the target physical distance information is used for reflecting the distance between a target vehicle and the target foreign matters;

a foreign object image acquisition module, configured to issue foreign object detection information to an image detection device and acquire to-be-processed foreign object image data detected by the image detection device based on the foreign object detection information, where the to-be-processed foreign object image data includes at least one frame of to-be-processed foreign object image of the target foreign object, when the target physical distance information is smaller than the reference physical distance information;

the image feature mining module is used for mining foreign object image feature representations of the foreign object image data to be processed by using a foreign object image analysis network, the foreign object image analysis network is formed by performing corresponding network optimization operation based on a first network learning cost index and a second network learning cost index, the first network learning cost index is a classified learning cost index, the second network learning cost index is other learning cost indexes except the classified learning cost index, and the first network learning cost index and the second network learning cost index are formed by performing corresponding analysis according to deep image feature representations of each foreign object image data;

The foreign object image classification module is used for analyzing corresponding foreign object image classification data based on the foreign object image characteristic representation and a reference foreign object image characteristic representation, the reference foreign object image characteristic representation is formed by mining the foreign object image data of the reference foreign object through the foreign object image analysis network, and the foreign object image classification data is used for reflecting the foreign object type information of the target foreign object;

and the foreign matter alarm module is used for carrying out foreign matter alarm operation based on the foreign matter image classification data.

In summary, the method and the system for alarming the falling foreign matters of the vehicle can obtain the target physical distance information of the target foreign matters; acquiring to-be-processed foreign matter image data detected by the image detection device based on the foreign matter detection information under the condition that the target physical distance information is smaller than the reference physical distance information; digging out a foreign object image characteristic representation of the foreign object image data to be processed by using a foreign object image analysis network; analyzing corresponding foreign object image classification data based on the foreign object image feature representation and the reference foreign object image feature representation; and performing a foreign object alarm operation based on the foreign object image classification data. Based on the foregoing, since the foreign object image analysis network performs the corresponding network optimization operation based on the first index of the network learning cost and the second index of the network learning cost to form the foreign object image analysis network, the first index of the network learning cost is the classified learning cost index, and the second index of the network learning cost is other learning cost indexes except the classified learning cost index, i.e. the network optimization operation is performed from two different dimensions, the accuracy of the formed foreign object image analysis network can be relatively high, the reliability of the foreign object image feature representation of the excavated foreign object image data to be processed can be ensured, the reliability of the foreign object image classified data analysis can be improved, the reliability of the foreign object alarm operation can be further ensured, and the defects of the prior art can be improved.

The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. The method for alarming the foreign matters falling from the vehicle is characterized by comprising the following steps:

2. The vehicle drop foreign object warning method according to claim 1, wherein the step of analyzing the corresponding foreign object image classification data based on the foreign object image feature representation and the reference foreign object image feature representation includes:

3. The vehicle drop foreign matter warning method according to any one of claims 1 or 2, characterized in that the vehicle drop foreign matter warning method further includes:

4. The method of claim 3, wherein the step of analyzing the corresponding target network learning cost index based on the foreign object image data corresponding to each of the representative foreign objects in the representative foreign object image data cluster comprises:

5. The method for warning of a vehicle of falling foreign objects according to claim 4, wherein the step of analyzing the corresponding first index of the network learning cost based on the deep image feature representation and the linear dimension importance parameter distribution corresponding to each of the foreign object image data comprises:

6. The method of claim 3, wherein the step of analyzing the corresponding target network learning cost index based on the foreign object image data corresponding to each of the representative foreign objects in the representative foreign object image data cluster comprises:

7. The method of claim 6, wherein the step of mining out deep image feature representations corresponding to each of the plurality of foreign object image data based on the foreign object image data corresponding to each of the plurality of the representative foreign object image data clusters using the convolutional network model with the gradient optimization operation, respectively, comprises:

8. The method of claim 6, wherein the step of analyzing the corresponding feature representation change parameters based on the deep image feature representation corresponding to each of the foreign object image data comprises:

extracting a predetermined optimization progress control hyper-parameter;

9. The method of claim 3, wherein the step of analyzing the corresponding target network learning cost index based on the foreign object image data corresponding to each of the representative foreign objects in the representative foreign object image data cluster comprises:

10. A vehicle drop foreign matter warning system, characterized by comprising a processor and a memory, the memory being for storing a computer program, the processor being for executing the computer program to implement the vehicle drop foreign matter warning method of any one of claims 1-9.