KR102479671B1 - Method for providing parts information of vehicle - Google Patents

Abstract

Disclosed is a method for providing information on parts of a vehicle performed by a computing device including at least one processor. The method may include the steps of: using a pre-trained vehicle part recognition module that takes a vehicle image as an input to identify a vehicle part in the vehicle image; generating identification information on parts of the vehicle from the vehicle image; and generating price information on each part for the vehicle image based on the vehicle part and the identification information.

Description

Method for providing vehicle parts information {METHOD FOR PROVIDING PARTS INFORMATION OF VEHICLE}

The present disclosure relates to the field of image processing, and more particularly, to a technique for providing part information and/or part-specific damage information of a damaged vehicle.

In the event of a vehicle accident, an offline method in which a damaged vehicle is moved to a construction company, and a repair estimate is presented by actually checking the vehicle within the construction company is common.

With the development of IT technology, an online platform is emerging that allows you to check a repair estimate through pictures of the vehicle and descriptions of the accident or accident part. Through this online platform, when damage or damage to a vehicle occurs, drivers can contact mechanics and ask for a repair estimate by delivering photos of the vehicle and explaining the details of the accident. In response to the request for a repair estimate, the mechanics check the vehicle picture and accident description, write the repair estimate directly, and deliver the estimate to the driver. come.

Even if it is possible to check the repair estimate more or less conveniently through the online platform, the degree of damage may be determined according to the subjective criteria of the vehicle expert at the repair shop, and accordingly, the repair cost estimate may differ significantly despite the similar degree of damage. It can happen often.

In addition, if the driver's description and photo data of the vehicle accident are judged according to the subjective criteria of each expert, there is a possibility that a difference may occur from the actual repair cost when the actual vehicle is received. may cause discomfort to

In addition, it may be a burden for general users to inquire about the price of the damaged parts when their vehicle is damaged. First, it may be difficult for general users to know how each part of the vehicle is divided. Second, even if it is possible to classify vehicle parts, it may be difficult to find information about a part code for a corresponding part, which is necessary for part price inquiry. This is because part codes must be identified through professional drawings produced by vehicle manufacturers.

Korean Patent Publication No. 10-2021-0025784

The present disclosure has been made in response to the foregoing background art, and is intended to identify damage for each vehicle part in a manner that efficiently uses computing resources. One embodiment of the present disclosure is to provide vehicle parts information in a manner that efficiently uses computing resources.

The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

According to some embodiments of the present disclosure for solving the above problems, a method for providing parts information of a vehicle performed by a computing device including at least one processor is disclosed. The method includes: using a pre-learned vehicle part recognition module that takes a vehicle image as an input, identifying a vehicle part in the vehicle image; generating identification information on parts of the vehicle from the vehicle image; and generating price information for each part of the vehicle image based on the vehicle part and the identification information.

In one embodiment, the vehicle part recognition module is an instance segmentation module capable of recognizing a boundary surface of the vehicle part, and a region for the vehicle part and a probability value corresponding to the region corresponding to the vehicle part. can output

In an embodiment, the vehicle part recognition module may output coordinate information about the part of the vehicle and quantitative information about a vehicle part class representing what part of the vehicle is in the vehicle image.

In one embodiment, the method may further include generating the vehicle image including information of a representative vehicle in the input image using a pretrained vehicle recognition module.

In one embodiment, the vehicle recognition module selects the representative vehicle by filtering vehicles less than a predetermined threshold area in the input image and the vehicle image including a boundary surface of the representative vehicle in the input image. can create

In an embodiment, the generating of the representative vehicle information may include determining a retake attempt when there is no vehicle recognized by the vehicle recognizing module; and when there are two or more vehicles recognized by the vehicle recognition module, a user's manual selection is induced by generating a bounding box including boundary surfaces of respective vehicles existing in the input image, or the input of the vehicles among the vehicles. A step of automatically determining a vehicle occupying the largest area in the image as a representative vehicle may be included.

In one embodiment, the vehicle part recognition module may be connected to the vehicle recognition module and receive an output of the vehicle recognition module to identify a vehicle part in the vehicle image.

In one embodiment, the method may further include generating a semantic feature for the captured image by using a pre-learned feature extraction module that receives a captured image of a vehicle captured by a user as an input. .

In one embodiment, the vehicle recognition module is connected to the output of the feature extraction module to receive the input image including the semantic feature and generate the vehicle image including representative vehicle information in the input image. can do.

In one embodiment, the feature extraction module is based on a self-supervised learning method before performing a common learning process that is learned together with at least one of the vehicle part recognition module and the vehicle recognition module. It is independently pre-trained, and the pre-training of the feature extraction module can be performed together with the decoder module.

In one embodiment, the feature extraction module generates a first semantic feature from a learning vehicle image in which a portion of the first image is masked, and the decoder module generates a second image corresponding to the first image from the first semantic feature , and the pre-learning may be performed based on a loss function between the first image and the second image.

In one embodiment, the method generates quantitative information about a damage type class representing a damage type of a vehicle in the vehicle image using a pre-learned damage type identification module that takes the vehicle image as an input. Further steps may be included.

In one embodiment, the damage type identification module is multi-label based for outputting a first probability value of a first damage type class and a second probability value of a second damage type class in units of pixels within the vehicle image. Semantic segmentation can be performed.

In an embodiment, the generating of price information for each part of the vehicle image may include generating price information for each part corresponding to a damaged part of the vehicle image based on the vehicle part, the identification information, and the damage type. steps may be included.

In one embodiment, generating price information for each part of the vehicle image includes: recognizing a license plate object included in the identification information and obtaining a vehicle number from the license plate object; extracting a vehicle identification number corresponding to the vehicle number; and obtaining metadata corresponding to the vehicle identification number by decoding the vehicle identification number. Here, the meta data may include at least one of manufacturer information, vehicle model information, vehicle body type information, transmission information, engine information, or production year information.

In an embodiment, the generating of price information for each part of the vehicle image may include generating parts information corresponding to a part of the vehicle and information on the number of parts required when replacing parts based on the metadata; The method may further include generating price information for each part based on the parts information and the number information of the parts.

In an embodiment, the identifying the vehicle part may include generating a mask map matrix for each part identifying the vehicle part in the vehicle image using the pretrained vehicle part recognition module. .

In one embodiment, the method may include determining an average area center coordinate of non-zero elements for the mask map matrix for each component; determining part price information corresponding to the average area center coordinate from the price information for each part; and generating a resultant image in which price information for the part is added in an augmented reality form to the average area center coordinates on a photographed image captured by a user.

In one embodiment, a computer program stored on a computer readable storage medium is disclosed. The computer program, when executed by one or more processors, performs a method for providing vehicle part information, the method using a pretrained vehicle part recognition module that takes a vehicle image as an input, identifying a vehicle part in; generating identification information on parts of the vehicle from the vehicle image; and generating price information for each part of the vehicle image based on the vehicle part and the identification information.

In one embodiment, a computing device for providing vehicle part information is disclosed. The computing device includes: one or more processors configured to: identify a vehicle part in the vehicle image using a pre-trained vehicle part recognition module that takes a vehicle image as an input; generating identification information for parts of the vehicle from the vehicle image; Further, price information for each part of the vehicle image may be generated based on the vehicle part and the identification information.

Techniques according to some embodiments of the present disclosure are intended to identify damage for each vehicle part in a manner that efficiently uses computing resources and/or to provide part information of a vehicle in a manner that efficiently uses computing resources.

Effects obtainable in the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below. .

Various aspects are now described with reference to the drawings, wherein like reference numbers are used to collectively refer to like elements. In the following embodiments, for explanation purposes, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it will be apparent that such aspect(s) may be practiced without these specific details.
1 illustratively illustrates a block diagram of a computing device in accordance with some embodiments of the present disclosure.
2 is a schematic diagram illustrating a network function according to some embodiments of the present disclosure.
3 illustratively illustrates a method for detecting a damaged area of a vehicle according to some embodiments of the present disclosure.
4 illustratively illustrates a method for detecting a damaged area of a vehicle according to some embodiments of the present disclosure.
5 schematically illustrates a vehicle recognition module according to some embodiments of the present disclosure.
6 schematically illustrates a post-processing module according to some embodiments of the present disclosure.
7 illustrates an operational relationship of modules for detecting a damaged area of a vehicle according to some embodiments of the present disclosure.
8 illustratively illustrates an operation according to the number of recognized vehicles according to some embodiments of the present disclosure.
9 illustratively shows priorities for each vehicle part according to an embodiment of the present disclosure.
10 illustrates an exemplary method for providing vehicle parts information according to one embodiment of the present disclosure.
11 illustratively illustrates operations of modules for providing parts information of a vehicle according to an embodiment of the present disclosure.
12 exemplarily illustrates a vehicle recognition module according to an embodiment of the present disclosure.
13 exemplarily illustrates a vehicle information extraction module according to an embodiment of the present disclosure.
14 illustratively illustrates a component information acquisition module according to an embodiment of the present disclosure.
15 illustratively illustrates information for each part of a vehicle generated by a vehicle part recognition module according to an embodiment of the present disclosure.
16 illustratively illustrates vehicle parts information generated according to an embodiment of the present disclosure.
17 depicts a simplified and general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is apparent that these embodiments may be practiced without these specific details.

The terms “component,” “module,” “system,” and the like, as used herein, refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure, processor, object, thread of execution, program, and/or computer running on a processor. For example, both an application running on a computing device and a computing device may be components. One or more components may reside within a processor and/or thread of execution. A component can be localized within a single computer. A component may be distributed between two or more computers. Also, these components can execute from various computer readable media having various data structures stored thereon. Components may be connected, for example, via signals with one or more packets of data (e.g., data and/or signals from one component interacting with another component in a local system, distributed system) to other systems and over a network such as the Internet. data being transmitted) may communicate via local and/or remote processes.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless otherwise specified or clear from the context, “X employs A or B” is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or, if X uses both A and B, "X uses either A or B" may apply to either of these cases. Also, the term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the listed related items.

Also, the terms "comprises" and/or "comprising" should be understood to mean that the features and/or components are present. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, elements, and/or groups thereof. Also, unless otherwise specified or where the context clearly indicates that a singular form is indicated, the singular in this specification and claims should generally be construed to mean "one or more".

In addition, the term “at least one of A or B” should be interpreted as meaning “when only A is included”, “when only B is included”, and “when A and B are combined”.

Those skilled in the art will further understand that the various illustrative logical blocks, components, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented using electronic hardware, computer software, or combinations of both. It should be recognized that it can be implemented as To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or as software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of this disclosure.

The description of the presented embodiments is provided to enable any person skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure. The general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention is to be accorded the widest scope consistent with the principles and novel features set forth herein.

1 illustratively illustrates a block diagram of a computing device in accordance with some embodiments of the present disclosure. For example, a technique for detecting damage to a vehicle and/or a technique for providing parts information of a vehicle described in this disclosure may be performed by the computing device 100 illustrated in FIG. 1 .

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device 100 may include other components for performing a computing environment of the computing device 100, and only some of the disclosed components may constitute the computing device 100. The computing device 100 in the present disclosure may refer to any type of device having processing capability. In one example, computing device 100 may be a server. As another example, the computing device 100 may be a user terminal. According to these examples, the technique for detecting damage to a vehicle and/or the technique for providing parts information of a vehicle described in this disclosure may be applied to a server and/or a user terminal encompassed by the computing device 100. can be performed by

Computing device 100 may include any type of computer system or computer device, such as, for example, a microprocessor, mainframe computer, digital processor, portable device or device controller, and the like. This computing device 100 may include any type of server and/or user terminal. For example, the user terminal may include a PC (personal computer), a notebook (note book), a mobile terminal, a smart phone (smart phone), a tablet PC (tablet pc), etc. owned by the user, , can include all types of terminals that can access wired/wireless networks.

The computing device 100 may include a processor 110 , a storage unit 120 , and a communication unit 130 .

The processor 110 may include one or more cores, and includes a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. unit), or any type of processor for data processing, data analysis, and/or deep learning. The processor 110 may read a computer program stored in the storage unit 120 and perform a method for identifying and evaluating damage for each part in a vehicle image according to an embodiment of the present disclosure. The processor 110 may read a computer program stored in the storage unit 120 and perform a method for providing part information of a damaged vehicle according to an embodiment of the present disclosure. In an additional embodiment, the processor 110 reads the computer program stored in the storage unit 120 to identify damage to each part in the vehicle image and to obtain part information (eg, price information for each part) corresponding to the damage to each part. Methods to provide can be performed.

According to an embodiment of the present disclosure, the processor 110 may perform an operation for learning a neural network. The processor 110 processes input data for learning in machine learning or deep learning (DL: preprocessing such as denoising of input data), extracts features from input data, calculates errors, and backpropagates. Calculations for neural network learning, such as weight update of the neural network using backpropagation, may be performed. At least one of the CPU, GPGPU, and TPU of the processor 110 may process learning of the network function. For example, the CPU and GPGPU can process learning of network functions and data classification using network functions. In addition, in an embodiment of the present disclosure, the learning of a network function and data classification using a network function may be processed by using processors of a plurality of computing devices together. In addition, the computer program executed in the computing device 100 according to an embodiment of the present disclosure may be a CPU, GPGPU or TPU executable program.

According to an embodiment of the present disclosure, the storage unit 120 may store any type of information generated or determined by the processor 110 and any type of information received by the communication unit 150. In the present disclosure, the storage unit 120 may be used interchangeably with a memory. For example, the storage unit 120 may store a first segmentation module, a second segmentation module, a feature extraction module, a decoder module, and/or a post-processing module. For example, the storage unit 120 may store a vehicle recognition module, a vehicle part recognition module, a damage type identification module, a post-processing module, and/or a feature extraction module. For example, the storage unit 120 may store a vehicle recognition module, a vehicle part recognition module, a vehicle information extraction module, a parts information acquisition module, and/or a parts information generation module.

According to an embodiment of the present disclosure, the storage unit 120 may be a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Memory) Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The computing device 100 may operate in relation to a web storage that performs the storage function of the storage unit 120 on the Internet. The above description of the storage unit 120 is only an example, and the present disclosure is not limited thereto.

The communication unit 130 according to an embodiment of the present disclosure may use any wired or wireless communication network capable of transmitting and receiving any type of data and signals. The communication unit 130 according to an embodiment of the present disclosure may communicate with any type of other computing device (eg, user terminal) through such a network.

A user terminal according to an embodiment of the present disclosure is a PC (personal computer), a notebook (note book), a mobile terminal (mobile terminal), a smart phone (smart phone), a tablet PC (tablet pc), etc. owned by a user. It may include all types of terminals capable of accessing wired/wireless networks.

2 is a schematic diagram illustrating a network function according to an embodiment of the present disclosure.

Throughout this specification, an artificial intelligence-based model, an artificial intelligence model, a computational model, a neural network, a network function, a neural network, or a pretrained module may be used interchangeably. For example, a segmentation module, a feature extraction module, a vehicle recognition module, a vehicle part recognition module, a post-processing module, and/or a damage type identification module in the present disclosure may be included in the category of the aforementioned artificial intelligence model. In a further example, the vehicle information extraction module, parts information acquisition module, and/or part information generation module in the present disclosure may also be included in the scope of the above-mentioned artificial intelligence model.

In this specification, a neural network may be composed of a set of interconnected computational units, which may be generally referred to as nodes. These nodes may also be referred to as neurons. A neural network includes one or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more links.

In a neural network, one or more nodes connected through a link may form a relative relationship of an input node and an output node. The concept of an input node and an output node is relative, and any node in an output node relationship with one node may have an input node relationship with another node, and vice versa. As described above, an input node to output node relationship may be created around a link. More than one output node can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of data of the output node may be determined based on data input to the input node. Here, a link interconnecting an input node and an output node may have a weight. The weight may be variable, and may be changed by a user or an algorithm in order to perform a function desired by the neural network. For example, when one or more input nodes are interconnected by respective links to one output node, the output node is set to a link corresponding to values input to input nodes connected to the output node and respective input nodes. An output node value may be determined based on the weight.

As described above, in the neural network, one or more nodes are interconnected through one or more links to form an input node and output node relationship in the neural network. Characteristics of the neural network may be determined according to the number of nodes and links in the neural network, an association between the nodes and links, and a weight value assigned to each link. For example, when there are two neural networks having the same number of nodes and links and different weight values of the links, the two neural networks may be recognized as different from each other.

A neural network may be composed of a set of one or more nodes. A subset of nodes constituting a neural network may constitute a layer. Some of the nodes constituting the neural network may form one layer based on distances from the first input node. For example, a set of nodes having a distance of n from the first input node may constitute n layers. The distance from the first input node may be defined by the minimum number of links that must be passed through to reach the corresponding node from the first input node. However, the definition of such a layer is arbitrary for explanation, and the order of a layer in a neural network may be defined in a method different from the above. For example, a layer of nodes may be defined by a distance from a final output node.

An initial input node may refer to one or more nodes to which data is directly input without going through a link in relation to other nodes among nodes in the neural network. Alternatively, in a relationship between nodes based on a link in a neural network, it may mean nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in relation to other nodes among nodes in the neural network. Also, the hidden node may refer to nodes constituting the neural network other than the first input node and the last output node.

In the neural network according to an embodiment of the present disclosure, the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as the number of nodes progresses from the input layer to the hidden layer. can In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes of the input layer may be less than the number of nodes of the output layer and the number of nodes decreases as the number of nodes increases from the input layer to the hidden layer. there is. In addition, the neural network according to another embodiment of the present disclosure is a neural network in which the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes increases as the number of nodes increases from the input layer to the hidden layer. can A neural network according to another embodiment of the present disclosure may be a neural network in the form of a combination of the aforementioned neural networks.

A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can reveal latent structures in data. That is, pictures, texts, videos, voices, textual data of voices, subliminal structure of music (e.g. what objects are in a picture, what is the content and emotion of the text, what is the content and emotion of the voice, etc. ) can be identified. Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, generative adversarial networks (GANs), and restricted boltzmann machines (RBMs). machine), a deep belief network (DBN), a Q network, a U network, a Siamese network, a Generative Adversarial Network (GAN), and the like. The description of the deep neural network described above is only an example, and the present disclosure is not limited thereto.

In a further embodiment of the present disclosure, the network function may include an autoencoder. An autoencoder may be a type of artificial neural network for outputting output data similar to input data. An auto-encoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between input and output layers. The number of nodes of each layer may be reduced from the number of nodes of the input layer to an intermediate layer called the bottleneck layer (encoding), and then expanded symmetrically with the reduction from the bottleneck layer to the output layer (symmetrical to the input layer). Autoencoders can perform non-linear dimensionality reduction. The number of input layers and output layers may correspond to dimensions after preprocessing of input data. In the auto-encoder structure, the number of hidden layer nodes included in the encoder may decrease as the distance from the input layer increases. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, a sufficient amount of information may not be conveyed, so more than a certain number (e.g., more than half of the input layer, etc.) ) may be maintained.

The neural network may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. Learning of the neural network may be a process of applying knowledge for the neural network to perform a specific operation to the neural network.

A neural network can be trained in a way that minimizes output errors. In the learning of the neural network, the learning data is repeatedly input into the neural network, the output of the neural network for the training data and the error of the target are calculated, and the error of the neural network is transferred from the output layer of the neural network to the input layer in the direction of reducing the error. It is a process of updating the weight of each node of the neural network by backpropagating in the same direction. In the case of teacher learning, the learning data in which the correct answer is labeled is used for each learning data (ie, the labeled learning data), and in the case of comparative teacher learning, the correct answer may not be labeled in each learning data. That is, for example, learning data in the case of teacher learning regarding data classification may be data in which each learning data is labeled with a category. Labeled training data is input to a neural network, and an error may be calculated by comparing an output (category) of the neural network and a label of the training data. As another example, in the case of comparative history learning for data classification, an error may be calculated by comparing input learning data with a neural network output. The calculated error is back-propagated in a reverse direction (ie, from the output layer to the input layer) in the neural network, and the connection weight of each node of each layer of the neural network may be updated according to the back-propagation. The amount of change in the connection weight of each updated node may be determined according to a learning rate. The neural network's computation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of iterations of the learning cycle of the neural network. For example, a high learning rate may be used in the early stage of neural network training to increase efficiency by allowing the neural network to quickly obtain a certain level of performance, and a low learning rate may be used in the late stage to increase accuracy.

As an example, in the present disclosure, a converted (or restored) vehicle image is generated from a learning vehicle image in which a portion of the original vehicle image is masked, and a loss function so that the difference between the generated vehicle image and the original vehicle image can be minimized can be updated. In the above manner, the feature extraction module may be pre-trained. As another example, the segmentation modules in the present disclosure may update a loss function such that a difference between an output value and a correct value (labeled value) is minimized.

In neural network learning, generally, training data can be a subset of real data (ie, data to be processed using the trained neural network), and therefore, errors for training data are reduced, but errors for real data are reduced. There may be incremental learning cycles. Overfitting is a phenomenon in which errors for actual data increase due to excessive learning on training data. For example, a phenomenon in which a neural network that has learned a cat by showing a yellow cat does not recognize that it is a cat when it sees a cat other than yellow may be a type of overfitting. Overfitting can act as a cause of increasing the error of machine learning algorithms. Various optimization methods can be used to prevent such overfitting. To prevent overfitting, methods such as increasing the training data, regularization, inactivating some nodes in the network during learning, and using a batch normalization layer should be applied. can

A computer readable medium storing a data structure according to an embodiment of the present disclosure is disclosed.

Data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. Data structure may refer to the organization of data to solve a specific problem (eg, data retrieval, data storage, data modification in the shortest time). A data structure may be defined as a physical or logical relationship between data elements designed to support a specific data processing function. A logical relationship between data elements may include a connection relationship between user-defined data elements. A physical relationship between data elements may include an actual relationship between data elements physically stored in a computer-readable storage medium (eg, a persistent storage device). The data structure may specifically include a set of data, a relationship between data, and a function or command applicable to the data. Through an effectively designed data structure, a computing device can perform calculations while using minimal resources of the computing device. Specifically, the computing device can increase the efficiency of operation, reading, insertion, deletion, comparison, exchange, and search through an effectively designed data structure.

The data structure can be divided into a linear data structure and a non-linear data structure according to the shape of the data structure. A linear data structure may be a structure in which only one data is connected after one data. Linear data structures may include lists, stacks, queues, and decks. A list may refer to a series of data sets in which order exists internally. The list may include a linked list. A linked list may be a data structure in which data are connected in such a way that each data is connected in a single line with a pointer. In a linked list, a pointer can contain information about connection to the next or previous data. A linked list can be expressed as a singly linked list, a doubly linked list, or a circular linked list depending on the form. A stack can be a data enumeration structure that allows limited access to data. A stack can be a linear data structure in which data can be processed (eg, inserted or deleted) at only one end of the data structure. The data stored in the stack may be a LIFO-Last in First Out (Last in First Out) data structure. A queue is a data listing structure that allows limited access to data, and unlike a stack, it can be a data structure (FIFO-First in First Out) in which data stored later comes out later. A deck can be a data structure that can handle data from either end of the data structure.

The nonlinear data structure may be a structure in which a plurality of data are connected after one data. The non-linear data structure may include a graph data structure. A graph data structure can be defined as a vertex and an edge, and an edge can include a line connecting two different vertices. A graph data structure may include a tree data structure. The tree data structure may be a data structure in which one path connects two different vertices among a plurality of vertices included in the tree. That is, it may be a data structure that does not form a loop in a graph data structure.

Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, a neural network is unified and described. The data structure may include a neural network. And the data structure including the neural network may be stored in a computer readable medium. The data structure including the neural network may also include preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation function associated with each node or layer of the neural network, and neural network It may include a loss function for learning of . A data structure including a neural network may include any of the components described above. That is, the data structure including the neural network includes preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation function associated with each node or layer of the neural network, and neural network. It may be configured to include all or any combination thereof, such as a loss function for learning of . In addition to the foregoing configurations, the data structure comprising the neural network may include any other information that determines the characteristics of the neural network. In addition, the data structure may include all types of data used or generated in the computational process of the neural network, but is not limited to the above. A computer readable medium may include a computer readable recording medium and/or a computer readable transmission medium. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network includes one or more nodes.

The data structure may include data input to the neural network. For example, the data structure may include data obtained as a result of one-hot encoding. A data structure including data input to the neural network may be stored in a computer readable medium. Data input to the neural network may include training data input during a neural network learning process and/or input data input to a neural network that has been trained. Data input to the neural network may include pre-processed data and/or data subject to pre-processing. Pre-processing may include a data processing process for inputting data to a neural network. Accordingly, the data structure may include data subject to pre-processing and data generated by pre-processing. The foregoing data structure is only an example, and the present disclosure is not limited thereto.

The data structure may include the weights of the neural network. (In this specification, weights and parameters may be used in the same meaning.) Also, a data structure including weights of a neural network may be stored in a computer readable medium. A neural network may include a plurality of weights. The weight may be variable, and may be changed by a user or an algorithm in order to perform a function desired by the neural network. For example, when one or more input nodes are interconnected by respective links to one output node, the output node is set to a link corresponding to values input to input nodes connected to the output node and respective input nodes. A data value output from an output node may be determined based on the weight. The foregoing data structure is only an example, and the present disclosure is not limited thereto.

As a non-limiting example, the weights may include weights that are varied during neural network training and/or weights for which neural network training has been completed. The variable weight in the neural network learning process may include a weight at the time the learning cycle starts and/or a variable weight during the learning cycle. The weights for which neural network learning has been completed may include weights for which learning cycles have been completed. Accordingly, the data structure including the weights of the neural network may include a data structure including weights that are variable during the neural network learning process and/or weights for which neural network learning is completed. Therefore, it is assumed that the above-described weights and/or combinations of weights are included in the data structure including the weights of the neural network. The foregoing data structure is only an example, and the present disclosure is not limited thereto.

The data structure including the weights of the neural network may be stored in a computer readable storage medium (eg, a memory or a hard disk) after going through a serialization process. Serialization can be the process of converting a data structure into a form that can be stored on the same or another computing device and later reconstructed and used. A computing device may serialize data structures to transmit and receive data over a network. The data structure including the weights of the serialized neural network may be reconstructed on the same computing device or another computing device through deserialization. The data structure including the weights of the neural network is not limited to serialization. Furthermore, the data structure including the weights of the neural network is a data structure for increasing the efficiency of operation while minimizing the resource of the computing device (for example, B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree). The foregoing is only an example, and the present disclosure is not limited thereto.

The data structure may include hyper-parameters of the neural network. Also, the data structure including the hyperparameters of the neural network may be stored in a computer readable medium. A hyperparameter may be a variable variable by a user. Hyperparameters include, for example, learning rate, cost function, number of learning cycle iterations, weight initialization (eg, setting the range of weight values to be targeted for weight initialization), hidden unit number (eg, the number of hidden layers and the number of nodes in the hidden layer). The foregoing data structure is only an example, and the present disclosure is not limited thereto.

At least some of the modules and/or models according to an embodiment of the present disclosure may operate based on a transformer. For example, a first segmentation module (eg, for identifying damage type) and a second segmentation module (eg, for identifying vehicle parts) may include a transformer-based feature extraction module to extract semantic features from an image. can utilize For example, the vehicle recognition module and/or the vehicle part recognition module may utilize a feature extraction module that generates semantic features for a captured image, and this feature extraction module may include a transformer.

A transformer may consist of an encoder encoding embedded data and a decoder decoding encoded data. The transformer may have a structure that receives a series of data and outputs a series of data of different types through encoding and decoding steps. In one embodiment, the series of data can be processed into a form operable by a transformer. A process of processing a series of data into a form in which a transformer can operate may include an embedding process. Expressions such as a data token, an embedding vector, and an embedding token may refer to embedded data in a form that can be processed by a transformer.

In order for the transformer to encode and decode a series of data, encoders and decoders within the transformer may be processed using an attention algorithm. The Attention Algorithm is to obtain the similarity of one or more keys for a given query, reflect the given similarity to each key and corresponding value, and then reflect the similarity to the value ( Values) may mean an algorithm that calculates an attention value by weighting.

Depending on how to set the query, key, and value, various types of attention algorithms can be classified. For example, when attention is obtained by setting the same query, key, and value, this may mean a self-attention algorithm. In order to process a series of input data in parallel, when the dimension of the embedding vector is reduced and individual attention heads are obtained for each divided embedding vector to obtain attention, this means a multi-head attention algorithm. can do.

In one embodiment, a transformer may consist of modules that perform a plurality of multi-head self-attention algorithms or multi-head encoder-decoder algorithms. In one embodiment, the transformer may also include additional elements other than the attention algorithm, such as an embedding layer, a normalization layer, and a softmax layer. A method of constructing a transformer using the attention algorithm may include a method disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated herein by reference.

Transformers can be applied to various data domains such as embedded natural language, segmented image data, and audio waveforms to convert a series of input data into a series of output data. In order to convert data having various data domains into a series of data that can be input to the transformer, the transformer can embed the data. Transformers can process additional data representing relative positional or phase relationships between a set of input data. Alternatively, a series of input data may be embedded by additionally reflecting vectors representing a relative positional relationship or phase relationship between input data to the series of input data. In one example, the relative positional relationship between a series of input data may include, but is not limited to, word order in a natural language sentence, relative positional relationship of each segmented image, temporal sequence of segmented audio waveforms, and the like. . A process of adding information representing a relative positional relationship or phase relationship between a series of input data may be referred to as positional encoding.

3 illustratively illustrates a method for detecting a damaged area of a vehicle according to some embodiments of the present disclosure.

The method of FIG. 3 may be performed by, for example, the computing device 100 of the present disclosure. The order of the steps illustrated in FIG. 3 is expressed as an example for convenience of description, and as an example, steps 320 and 330 may be performed in parallel with each other.

The computing device 100 may extract a semantic feature of the vehicle image by using a pretrained feature extraction module that takes the vehicle image as an input (310).

In one embodiment of the present disclosure, a vehicle image may refer to any type of image including at least a portion of a vehicle. As an example, the vehicle image may include a 2D or 3D image including the exterior of the entire vehicle. As another example, the vehicle image may include a 2D or 3D image including a part of the vehicle.

In order to increase the efficiency and accuracy of learning of other modules (eg, the first segmentation module and/or the second segmentation module) for identifying a part of a vehicle or recognizing a damage type of a vehicle, the computing device 100 A semantic feature (semantic feature) of an input vehicle image may be extracted. The semantic feature may refer to any type of data expressing a feature of an input image. As an example, the semantic feature may mean a feature map corresponding to an input image.

In one embodiment of the present disclosure, the feature extraction module may mean an artificial intelligence-based model that can be learned through a self-supervised learning process. As described above, the feature extraction module is a module that outputs a vector or the like for expressing the feature of the input vehicle image from the input vehicle image. This feature extraction module may include, for example, a Convolution Neural Network (CNN), and in this example, a feature map may be output by the feature extraction module. The feature extraction module may perform a common learning process that is learned together with other modules (eg, the first segmentation module and/or the second segmentation module) for identifying parts of the vehicle or recognizing damage types of the vehicle. The feature extraction module may be independently pre-learned based on the above-described self-supervised learning method before performing such a common learning process.

The computing device 100 identifies a type of damage in the vehicle image using a first pretrained segmentation module that takes the semantic feature as an input (320) and uses a second pretrained segmentation module that takes the semantic feature as an input It is possible to identify a vehicle part in the vehicle image using (330).

In one embodiment, steps 320 and 330 may be performed sequentially with each other. For example, step 320 is performed first and step 330 is performed subsequently, or vice versa. In other embodiments, steps 320 and 330 may be performed in parallel or independently.

In one embodiment of the present disclosure, the first segmentation module may identify the type (type) of damage or damage within the vehicle image. In one embodiment, the second segmentation module may recognize or identify vehicle parts within the vehicle image. In an embodiment of the present disclosure, a feature extraction module commonly used for the first segmentation module and the second segmentation module instead of learning a separate image feature extraction module for each of the first segmentation module and the second segmentation module. this can be utilized. In one embodiment, the first segmentation module and the second segmentation module may use the output generated by the feature extraction module as an input. In a further embodiment, not only the output of the last layer of the feature extraction module but also the output of the middle layer may be passed to the first segmentation module and the second segmentation module.

In one embodiment, the first segmentation module may output first quantitative information about a damage type class indicating a damage type in the vehicle image. For example, there may be a plurality of damage type classes indicating what type of damage has occurred in the vehicle image. The first segmentation module may output quantitative information indicating a possibility of corresponding to a corresponding class for each of a plurality of damage type classes or for some of the plurality of damage type classes.

In one embodiment, the first segmentation module performs multi-label based semantic segmentation for outputting a first probability value of a first damage type class and a second probability value of a second damage type class in units of pixels within a vehicle image. can be done The first segmentation module may output a segmentation map for each vehicle damage.

For example, the first segmentation module may output scores for classes indicating types of damage, such as “painting peeled off”, “broken”, “dented”, and/or “scratched” with respect to the vehicle image. As another example, the first segmentation module may determine specific parts of a vehicle, such as a class corresponding to “front bumper deformed”, a class corresponding to “front bumper paint peeling”, a class corresponding to “front passenger seat fender broken”, etc. Quantitative information on each of the classes matched with the damage type may be output.

In one embodiment of the present disclosure, the second segmentation module may output information for identifying parts or parts of a vehicle included in a vehicle image. For example, the second segmentation module may include coordinate information about a vehicle part present in the vehicle image, coordinate information about a vehicle boundary surface present in the vehicle image, and/or a vehicle part class indicating what the vehicle part is in the vehicle image. Quantitative information about can be output. The second segmentation module identifies one or more instances of vehicle parts by performing instance segmentation with semantic features including image feature maps output from the middle layer and/or the last layer of the feature extraction module as input. can do. For example, the second segmentation module may, for a vehicle image, a front door, a rear door, a bonnet, a trunk, a headlight, a side mirror, a tail lamp mounted on a trunk side, a tail lamp mounted on a quarter panel side, a wheel, and a front A class corresponding to the vehicle image may be determined as at least one class among a plurality of classes including a bumper, a rear bumper, a front fender, a rear fender, a side step, a grill, and/or an entire vehicle. That is, the second segmentation module may divide and recognize the part of the input vehicle image. The second segmentation module may indicate a probability of an instance for each part of the vehicle or a value for an instance for each part in a binary matrix composed of 0 and 1. For example, the second segmentation module may output a segmentation map for each part of the vehicle.

In one embodiment of the present disclosure, computing device 100 may generate a damage result for the vehicle image based at least in part on the type of damage and the location of the vehicle ( 340 ).

In one embodiment, the damage result for the vehicle image may include the type of damage to each part of the vehicle and/or the intensity of damage to each part of the vehicle. In an additional embodiment, the damage result for the vehicle image may further include repair cost information generated as a result of damage for each part of the vehicle.

In an embodiment, the computing device 100 applies a predetermined priority value for each vehicle part to a segmentation map for each vehicle part that is an output of the second segmentation module, using the post-processing module, so that the vehicle including the priority value A vehicle projection image including the parts may be generated. As an example, a priority value may be pre-assigned as 112 for the front fenders, 80 for the front bumper, 160 for the headlights, and/or 208 for the wheels. The post-processing module may output a quantitative value in which the priority value is reflected for each part of the vehicle by combining the output result of the second segmentation module using the priority value.

In one embodiment, the computing device 100 includes a vehicle projection image to which vehicle damage segmentation is applied by merging a segmentation map for each vehicle damage, which is an output of the first segmentation module, with the projected vehicle image using a post-processing module. Can produce damaging results. The post-processing module combines the vehicle projection image generated using the output result of the second segmentation module and the priority value with the output result of the first segmentation module to determine not only the damaged part on the vehicle projection image, but also the intensity or type of the damaged part. can be printed out.

As described above, since the segmentation module for recognizing vehicle damage and the segmentation module for recognizing vehicle parts share a feature extraction module, the technique according to an embodiment of the present disclosure uses the same amount of training datasets. Even if the artificial neural network is learned, it can have higher accuracy in damage recognition and part recognition than in the case where the feature extraction module is not shared. In addition, the technique according to an embodiment of the present disclosure shares a single feature extraction module without generating two different feature extraction modules corresponding to each of a segmentation model for vehicle damage recognition and a segmentation module for vehicle part recognition. Therefore, technical effects can be achieved in that not only the amount of memory used in the computing device during the learning process and the inference process can be reduced by about half, but also the amount of computation in the learning process and the inference process can be reduced by about half. Performance improvement from a hardware point of view can be derived according to the above-described technical effects.

4 illustratively illustrates a method for detecting a damaged area of a vehicle according to some embodiments of the present disclosure.

In one embodiment, the method illustrated in FIG. 4 may be performed by computing device 100 . The computing device 100 may include a feature extraction module 420 , a first segmentation module 430 , a second segmentation module 440 , a vehicle recognition module 500 and a post-processing module 600 .

When the vehicle image 410 is received, the feature extraction module 420 may extract semantic features for the vehicle image 410 .

In one embodiment, for the learning of the vehicle part identification module (second segmentation module) and the vehicle damage type recognition module (first segmentation module), the main semantic features need to be extracted in advance from the input image. There is a need. In order to extract these semantic features, a CNN (eg, ResNet, RegNet, EfficientNet, etc.) or transformer (eg, ViT, Swin-Transformer, Deformable-Transformer, etc.)-based feature extraction module 420 may be utilized. there is.

In one embodiment of the present disclosure, one feature extraction module is configured, and the output of this one feature extraction module is a second segmentation module 440 for vehicle part recognition and a first segmentation module for vehicle damage recognition. 430, and can be used as an input of the vehicle recognition module 500.

In one embodiment, the feature extraction module 420 is commonly used in an inference process simply by the second segmentation module 440 for vehicle part recognition and the first segmentation module 430 for vehicle damage recognition. Rather, it can be learned by being connected with the above-described modules in the artificial neural network learning process within the overall architecture.

The feature extraction module 420 of the present disclosure is separately separated prior to the learning process in which the first segmentation module 430 and the second segmentation module 440 are trained together, and the feature of the vehicle image is better for a large amount of vehicle image dataset. It can be trained alone for extraction.

In one embodiment, the pre-learning of the feature extraction module 420 may be performed based on a self-supervised learning method. For example, pretraining of the feature extraction module 420 may be performed together with the decoder module. The feature extraction module 420 generates a first semantic feature from a learning vehicle image in which a portion of the first original vehicle image is masked, and the decoder module generates a first semantic feature corresponding to the first original vehicle image from the first semantic feature. 2 original vehicle images may be restored, and pretraining of the feature extraction module 420 may be performed based on a loss function between the first original vehicle image and the second original vehicle image.

In one embodiment, a self-supervised learning method may be utilized as an example of a method of prior learning. For example, the self-supervised learning method according to an embodiment of the present disclosure may divide a given vehicle image 410 into n×m rectangular regions. Here, n and m may mean natural numbers. When each segmented area is defined as a patch, the computing device 100 removes some information (for example, marking it as black) for the number of patches as much as X% of the total number of n x m patches, and then the image feature extraction module 420 ) into the Here, X may mean a rational number greater than zero. The self-supervised learning method uses a feature extraction module 420 and a decoder module based on a loss function that restores an image masked by a patch to an original image by a decoder module composed of an Inverse CNN artificial neural network or Transformer-based architecture. learn together In an example according to an embodiment of the present disclosure, each of n x m may be 14 and X may be 75%, but values of n, m, and X may be variably adjusted according to implementation aspects.

In an embodiment, the second segmentation module 440 may perform coordinate information on a vehicle part present in the vehicle image 410 , coordinate information on a vehicle boundary present in the vehicle image 410 , and vehicle in the vehicle image 410 . Quantitative information about the vehicle part class indicating what part is can be output. In addition, the second segmentation module 440 performs instance segmentation by taking the semantic feature including the image feature map output from the middle layer and/or the last layer of the feature extraction module 420 as an input, thereby obtaining one or more vehicle parts. instance can be identified.

For example, the second segmentation module 440 for recognizing vehicle parts receives information (eg, an image feature map) output from the middle layer and/or the last layer of the feature extraction module 420 as an input, and obtains a vehicle image. In operation 410, instance segmentation may be performed to output coordinate information of a vehicle part and a vehicle boundary surface existing in the vehicle part and a probability of belonging to the vehicle part or vehicle category. This second segmentation may include, by way of non-limiting example, Mask R-CNN and/or Cascade Mask R-CNN modules. The second segmentation module 440 of the present disclosure includes, for example, a front door, a rear door, a bonnet, a trunk, a headlight, a side mirror, a tail lamp mounted on a trunk side, a tail lamp mounted on a quarter panel side, and a wheel. , Front bumper, rear bumper, front fender, rear fender, side step, grill, and the vehicle image 410 as a whole can be divided and recognized. The second segmentation module 440 may output probability segmentation for instances for each part of the vehicle or output a binary matrix composed of 0 or 1.

In one embodiment of the present disclosure, the second segmentation module 440 may operate based additionally on the output of the vehicle recognition module 500 . For example, the second segmentation module 440 may receive semantic features and representative vehicle information as inputs.

In one embodiment of the present disclosure, the computing device 100 may determine a representative vehicle within the vehicle image 410 using the pretrained vehicle recognition module 500 that takes the semantic feature as an input. In one embodiment, the vehicle recognition module 500 may determine a representative vehicle within the vehicle image 410 by filtering vehicles less than a predetermined threshold area within the vehicle image 410 . For example, the vehicle recognizing module 500 may determine re-photographing attempts when there is no recognized vehicle. When there are two or more recognized vehicles, the vehicle recognition module 500 induces a user's manual selection by generating a bounding box including boundary surfaces of respective vehicles existing in the vehicle image 410, or Among them, a vehicle occupying the largest area in the vehicle image 410 may be automatically determined as the representative vehicle.

As described above, the vehicle recognition module 500 may refer to a module for recognizing or selecting a vehicle intended by a user in the vehicle image 410 . That is, the vehicle recognition module 500 may determine a representative vehicle from the vehicle image 410 . The vehicle recognition module 500 may select a vehicle in the vehicle image 410 by receiving the vehicle image 410 and/or the semantic feature output from the feature extraction module 420 . The output of the vehicle recognition module 500 (eg, information on the selected vehicle image, etc.) is input to the first segmentation module 430 and the second segmentation module 440, and the first segmentation module 430 and the second segmentation The accuracy of the operation of the module 440 may be increased. A detailed description of the vehicle recognition module 500 will be described later with reference to FIG. 5 .

In one embodiment of the present disclosure, the first segmentation module 430 for recognizing vehicle damage receives the output of the feature extraction module 420 and/or the output of the vehicle recognition module 500 as inputs, and the vehicle image 410 ), the type of damage in can be output. For example, the first segmentation module 430 may receive semantic features and representative vehicle information as inputs.

In one embodiment, for example, the first segmentation module 430 may be executed when the vehicle recognition module 500 determines that a single vehicle is present. In one embodiment, the first segmentation module 430 may output quantitative information about a damage type class indicating a damage type in the vehicle image 410 . For example, the first segmentation module 430 takes information (eg, an image feature map) output from the middle and/or last layer of the feature extraction module 420 as an input and determines whether or not each pixel is broken, distorted, scratched, and/or broken. Alternatively, it is a semantic segmentation module that outputs whether or not separation exists as a probability value between 0 and 1.

In one embodiment, the first segmentation module 430 outputs a first probability value of a first damage type class and a second probability value of a second damage type class in units of pixels within the vehicle image 410. label)-based semantic segmentation can be performed. For example, the first segmentation module 430 does not classify (multi-class) each pixel of the final semantic segmentation map as one of the above-listed damage types, but rather classifies each damage type generated in the corresponding pixel. All probability values can be output (multi-label).

According to an embodiment of the present disclosure, multiple impairments can be detected using one module (eg, the first segmentation module 430) without the need to separately develop individual semantic segmentation artificial neural networks to detect multiple impairments. For example, dents and scratches are damages that frequently occur simultaneously, and in the output result of a pixel in which both damages occur together, a value with a probability value close to 1 can be output for both damages. In order to output a probability value, in general, as many binary semantic segmentation modules as the number of damage types need to be separately trained.

In an embodiment of the present disclosure, since the first segmentation module 430 is designed to share up to the artificial neural network layer at the front stage except for the last layer that outputs the occurrence probability value for each damage type, a plurality of types of damage In predicting the probability value for each type, the technical effect of reducing the amount of computation, delay time, and learning time can be derived. For example, in the last layer of the first segmentation module 430, a matrix having a dimension of [W x H x N] is output, and a sigmo is generated for each [W x H] two-dimensional matrix generated along the N dimension. A probability value between 0 and 1 may be output by taking a sigmoid function. The value is used to adjust the weight of the model in the learning process in the direction of minimizing the correct answer annotation result and error by the L1 or L2 loss function for each pixel. Here, W: the number of horizontal pixels, H: the number of height pixels, and N: the number of damage types. The resolution size of the Semantic Segmentation Map output from the first segmentation module 430 is not limited to the same structure as the input image, and the structure can have a smaller resolution size according to the computational power or delay time of the hardware provided in the operating environment. can be changed.

In one embodiment of the present disclosure, the post-processing module 600 may generate the result information 470 based at least in part on the output of the first segmentation module 430 and/or the output of the second segmentation module 440. there is.

In one embodiment, the post-processing module 600 applies a predetermined priority value for each vehicle part to the segmentation map for each vehicle part, which is an output of the second segmentation module 440, so that the vehicle parts including the priority value are selected. It is possible to generate a vehicle projection image that includes. For example, the post-processing module 600 generates a median value for each vehicle part by multiplying a matrix for each vehicle part corresponding to the segmentation map for each vehicle part by a priority value for each vehicle part, and The projected vehicle image having a unique score for each vehicle part may be generated by adding a background matrix including a zero matrix to the median value of each vehicle part in order according to the magnitude of the priority value for each part.

In one embodiment, the post-processing module 600 merges the segmentation map for each vehicle damage, which is an output of the first segmentation module 430, with the projected vehicle image, and results information including the projected vehicle image to which the vehicle damage segmentation is applied. (470). For example, the post-processing module 600 adds up a unique score for each vehicle part of the vehicle projection image to a matrix for each vehicle damage corresponding to the segmentation map for each vehicle damage, which spans a plurality of vehicle parts. Result information 470 identifying vehicle damage may be generated. A detailed description of the post-processing module 600 will be described later with reference to FIG. 6 .

5 schematically illustrates a vehicle recognition module 500 according to some embodiments of the present disclosure.

As shown in FIG. 5 , the vehicle recognition module 500 may include a vehicle instance filtering module 510 and a vehicle selection module 520 .

In one embodiment, vehicle instance filtering module 510 may determine what a representative vehicle is within vehicle image 410 by filtering vehicles that are less than a predetermined threshold area within vehicle image 410 . As an example, the vehicle instance filtering module 510 may perform an operation to remove vehicles having a specific critical area or less from the vehicle image 410 . The vehicle instance filtering module 510 may remove vehicles occupying an area smaller than a threshold of, for example, 0.2, on the segmentation map for all recognized vehicle instances. The vehicle instance filtering module 510 may be operated by any type of image processing technique to compare the sizes of vehicles and remove vehicles under a certain size. Regarding the aforementioned threshold value, it was desirable to experimentally filter out vehicles occupying an area smaller than the threshold value of 0.2, and the value is merely an embodiment described for explanatory purposes and may be variably adjusted according to the operational purpose of the system. there is.

In a further embodiment of the present disclosure, the vehicle instance filtering module 510 may be selectively operated. For example, the vehicle instance filtering module 510 may automatically remove a vehicle having a size smaller than a specific area from the image, but whether or not to operate may be determined according to a user's selection. According to an implementation aspect, the vehicle recognition module 500 provides the user with a situation that a plurality of vehicles exist, and the operation of the vehicle instance filtering module 510 may be replaced by manual selection by the user.

In one embodiment, vehicle recognition module 500 may include vehicle selection module 520 . Referring to FIG. 8 with respect to the vehicle selection module 520 . 8 illustratively illustrates an operation according to the number of recognized vehicles according to some embodiments of the present disclosure. The operations of FIG. 8 may be performed by the computing device 100 . For example, the operations of FIG. 8 may be performed by vehicle selection module 520 .

When a plurality of vehicles exist in the vehicle image 410, a result of performing damage and part recognition on all vehicles regardless of a specific vehicle may be output. For this reason, when a user inserts an image captured in a space where cars are densely populated (eg, a parking lot) as an input image, there is a possibility that the user's intention and the recognition result of a different vehicle may be conveyed to the user. In addition, in a situation where there is no separate filter unit for the vehicle image 410, a result related to the vehicle may be output when the vehicle does not exist in the photographed picture. For this reason, even if the user shoots an object other than the vehicle, there is a possibility that the non-vehicle area may be misrecognized, and if the vehicle does not exist in the vehicle image 410, it is not simply that there is no recognition result, but that the vehicle has not been recognized, so re-shooting is recommended. It is desirable to deliver a notification that In addition, even if the vehicle is recognized, if the size is too small, the accuracy and reliability of the vehicle part or damage recognition result may be significantly reduced, so it may be regarded as not a normally photographed vehicle image.

Therefore, the vehicle selection module 520 according to an embodiment of the present disclosure may perform a vehicle even when a plurality of vehicles are recognized through the algorithm presented in FIG. 8 for the recognized individual vehicles or in a situation where no vehicle is recognized. It may allow to efficiently perform a technique for recognizing damage by part and/or providing part information of a vehicle.

In one embodiment of the present disclosure, the vehicle selection module 520 may operate in two major modes according to a method preset by a system operator. In one embodiment, a vehicle selection module 520 may be devised that governs branching of the system according to the number of individual vehicles filtered out by the vehicle instance filtering module 510 .

In one embodiment of the present disclosure, when the number of recognized vehicles is 0 (810a), it is assumed that there is no vehicle on the vehicle image 410, and the user terminal side displays 'There is no recognized vehicle. Would you like to do a reshoot?' A query 820 may be generated. If the user selects yes, it is possible to allow the user to select a new input image or to re-photograph (830). The recaptured image may be re-input into the feature extraction module 420 . If the user selects No, the operation according to an embodiment of the present disclosure may end (840).

In one embodiment of the present disclosure, when the number of recognized vehicles is one (810b), the vehicle selection module 520 transfers corresponding information to the first segmentation module 420 and/or the second segmentation module 430 without any special processing. can deliver.

In an additional embodiment of the present disclosure, when the number of recognized vehicles is 1 (810b), the vehicle selection module 520 performs the first segmentation module 420 and/or the second segmentation with the post-processing module 600 without any special processing. You may also forward the results of module 430. In this embodiment, the vehicle recognition module 500 may have output values of the first segmentation module 420 and/or the second segmentation module 430 .

In one embodiment of the present disclosure, when the number of recognized vehicles is two or more (810c), the vehicle selection module 520 may request manual selection (850) or automatically select (860) a representative vehicle by the user. can

In one embodiment, in the manual selection 850 mode, the vehicle selection module 520 generates a user interface (UI) for displaying a plurality of recognized vehicles to the user terminal and allowing a single vehicle to be recognized to be selected. can In one embodiment, the UI may include a UI screen in which a boundary surface of a vehicle or a bounding box including the boundary surface is overlaid on an input image. In one embodiment, the UI may include a UI screen displaying a list of cropped vehicle images for each bounding box including the recognized vehicle boundary surface. For example, the aforementioned bounding box may mean a box having a minimum size that may include a boundary surface of a vehicle.

In one embodiment, vehicle selection module 520 may utilize the output of vehicle instance filtering module 510 . For example, in the representative vehicle automatic selection 860 mode, the vehicle selection module 520 selects a vehicle that occupies the largest portion of the area of the entire image among individual vehicles from the output result of the vehicle instance filtering module 510 as the representative vehicle. considered and selected automatically.

In one embodiment, in the manual selection mode 850, since the user needs to be requested to select a representative vehicle, unnecessary user intervention and recognition delay time may occur. Accordingly, the vehicle selection module 520 may automatically determine a vehicle to be analyzed within the vehicle image 410 without user intervention by automatically selecting a representative vehicle from the output result of the vehicle instance filtering module 510. there is.

In an additional embodiment of the present disclosure, the vehicle instance filtering module 510 may be operated by receiving an operation result of the vehicle selection module 520 . For example, the vehicle instance filtering module 510 may automatically remove vehicle part instances existing in an area other than the area of the representative vehicle selected from the vehicle selection module 520. For example, the vehicle instance filtering module 510 ) performs an element-wise multiplication operation with the mask map of the representative vehicle for each vehicle part instance, so that the ratio of the intersection area to the total area is lower than a specific threshold value. Instances can be removed. Such an operation may include a matrix multiplication operation and a vector operation, and thus may have an advantage of reducing latency when implemented by a processing unit capable of parallel operation (eg, GPU, TPU, IPU, etc.).

6 schematically illustrates a post-processing module 600 according to some embodiments of the present disclosure.

In one embodiment of the present disclosure, the post-processing module 600 may include a vehicle part projection module 610, a damage segmentation module 620 for each part, and a damage boundary surface extraction module 630.

The vehicle part projection module 610 takes as input vehicle part instance information from which only vehicle parts belonging to a representative vehicle are selected, and replaces the matrix value with a priority value for each part in a single matrix having a size of, for example, [W x H]. can do. The vehicle part projection module 610 may apply a predetermined priority value for each vehicle part illustrated in FIG. 9 to the segmentation map for each vehicle part, which is an output of the second segmentation module 440 . Here, W and H may mean rational numbers greater than zero.

Regarding the predetermined priority value for each vehicle part, referring to FIG. 9 , a mapping table 900 in which the predetermined priority value 910 is mapped in units of vehicle part names 920 is described as an example. The vehicle part projection module 610 may generate a vehicle projection image having a unique score for each vehicle part by referring to the mapping table 900 .

The vehicle part projection module 610 may generate a vehicle projection image including vehicle parts to which priority values are applied. For example, the vehicle part projection module 610 multiplies a matrix for each vehicle part corresponding to the segmentation map for each vehicle part, which is an output of the second segmentation module 440, by a priority value for each vehicle part illustrated in FIG. The projection of the vehicle having a unique score for each vehicle part by generating a median value for each part and adding a background matrix including a zero matrix to the median value for each part of the vehicle in an order according to the magnitude of the priority value for each part of the vehicle. image can be created.

For example, the vehicle part projection module 610 first initializes a [W x H] background matrix filled with 0 values (ie, a 0 matrix). Next, the vehicle part projection module 610 multiplies the segmentation matrix for each part consisting of 0 or 1 by the priority value for each part and adds it to the background matrix. That is, the vehicle part projection module 610 may utilize the segmentation matrix for each part of the vehicle obtained from the second segmentation module 440 . The segmentation matrix for each part of the vehicle may include one corresponding matrix for each specific part of the vehicle, and may include a corresponding matrix for each of a plurality of parts. In this matrix, a value corresponding to 1 may exist at a position corresponding to a specific part, and a value corresponding to 0 may exist at a position of a different part. The vehicle part projection module 610 may add a value multiplied by a priority value for each matrix for each part of the vehicle with a background matrix. Accordingly, a projected vehicle image having a unique score for each vehicle part may be generated.

In an embodiment, the vehicle part projection module 610 may perform the summation with the background matrix in order of projection priority for each preset vehicle part. In this way, if projection priority is not specified for the background matrix, for example, if the vehicle's headlight region is added to the background matrix ahead of the front bumper region, the front bumper region usually contains the headlight region and Therefore, the entire headlight area can be replaced with the front bumper area. Accordingly, the vehicle part projection module 610 according to an embodiment of the present disclosure may perform summation with the background matrix prior to headlights for, for example, a front bumper in order of projection priority for each preset vehicle part.

More specifically, the lower the number in the priority order for each part as described in the mapping table 900 in FIG. 9 , the higher the priority value may be added to the background matrix. The priority values presented in FIG. In the case of having a region and including a region of another region, the value may be changed according to an implementation mode as long as the condition that a region having a larger region is given a lower priority is satisfied.

In an embodiment of the present disclosure, a projected vehicle image having a unique score for each vehicle part may refer to a vehicle image in which a specific score is displayed for each part of the vehicle.

According to an embodiment of the present disclosure, a damage segmentation module 620 for each part is disclosed. This part-specific damage division module can perform the function of a damage splitter. Extensive damage can occur over a plurality of vehicle parts, and it is necessary to separate the damaged area for each part in order to predict vehicle repair costs and damage.

In the case of detecting damage using the object detection method, the result will be output in the form of a bounding box that includes the damage, and since this bounding box only has information on the top, bottom, left, and right edges of the damaged area, it can be applied to a plurality of vehicle parts. If damage occurs over time, it may have a limitation that it is impossible to accurately divide the damage bounding box.

In an embodiment of the present disclosure, the damage segmentation module 620 for each part may divide the vehicle part segmentation map and the segmentation information for each damage type through element-wise summation. The part-specific damage segmentation module 620 determines the damaged area of the vehicle part based on the segmentation information for each damage type output from the first segmentation module 430 and the vehicle part segmentation map output from the second segmentation module 440. can be distinguished. Matrix sum operations in element units may be required as many times as the number corresponding to the types of supported damage types. In one embodiment, if four damage types are supported, all vehicle damage areas for all vehicle parts may be identified by performing a matrix sum operation four times.

In an additional embodiment of the present disclosure, the part-by-part damage segmentation module 620 performs element-wise analysis between segmentation information for each damage type, which is an output of the first segmentation module 430, and a vehicle projection image, which is an output of the vehicle part projection module 610. By performing the summation, it is possible to generate a damage segmentation map divided according to vehicle parts for each damage type. A matrix sum operation in element units between the vehicle projection image and segmentation information for each damage type may be performed as many times as the number of damage types. Accordingly, a priority value applied to the projected vehicle image for the damaged region is obtained through element-wise summation between the segmentation information for each damage type, which is an output of the first segmentation module 430, and the projected vehicle image, which is an output of the vehicle part projection module 610. An additional value such as 1 may be added to this reflected value. That is, the damage segmentation module 620 for each part converts the vehicle part of the projection image of the vehicle that is the output of the vehicle part projection module 610 to the matrix for each vehicle damage corresponding to the segmentation map for each vehicle damage that is the output of the first segmentation module 430 . By summing the unique scores per star, vehicle damage across multiple vehicle parts can be identified. For example, a front door of a vehicle may have a priority value of 16 in the mapping table 900 of FIG. 9 . The rear door of the vehicle may have a priority value of 32 in the mapping table 900 of FIG. 9 . When damage occurs across the front and rear doors of the vehicle, the area included in the front door among the damaged areas has a value of 17, and the area included in the rear door among the damaged areas has a value of 33. can have

As described above, a damage segmentation map divided according to vehicle parts for each damage type may be generated through the damage segmentation module 620 for each part.

In one embodiment, a damage interface extraction module 630 is disclosed. The damage boundary surface extraction module 630 may refer to a module for extracting a list of coordinates of the boundary surface for each damaged region by using the output result of the damage segmentation module 620 for each part as an input. The damage boundary surface extraction module 630 obtains a list of boundary lines based on an arbitrary contour search algorithm after filtering pixels according to the code area for each part with respect to the segmentation map generated for each vehicle part and each damage type. can do. Accordingly, the computing device 100 may extract from where the damage occurs depending on the part of the vehicle through a more accurate boundary line.

According to the operation of the post-processing module 600 according to an embodiment of the present disclosure, even damage or breakage occurring over a plurality of parts can be accurately expressed in the vehicle image.

7 illustrates an operational relationship of modules for detecting a damaged area of a vehicle according to some embodiments of the present disclosure.

As shown in FIG. 7 , the computing device 100 may receive a vehicle image 710 photographed by a user. The vehicle image 710 may represent, for example, an image of a vehicle including a damaged or damaged area.

In one embodiment, the feature extraction module 420 generates semantic features for the input vehicle image 710 and sends them to the first segmentation module 430, the second segmentation module 440 and the vehicle recognition module 500. can be conveyed In one embodiment of the present disclosure, the semantic feature may include a semantic feature map. In a further embodiment of the present disclosure, the semantic features and vehicle image 710 may be passed together to the first segmentation module 430 , the second segmentation module 440 and the vehicle recognition module 500 .

In one embodiment, the vehicle recognition module 500 converts vehicle selection result information to be utilized in the first segmentation module 430 and the second segmentation module 440 based on the semantic features generated by the feature extraction module 420. can create For example, information other than the vehicle selected from the vehicle image 710 may be removed from the vehicle selection result information.

In one embodiment, the first segmentation module 430 may output a vehicle damage recognition result 730 based on the output of the feature extraction module 420 and the output of the vehicle recognition module 500 . Additionally, the first segmentation module 430 may output a vehicle damage recognition result 730 based on the vehicle image 710 additionally.

The vehicle damage recognition result 730 may mean, for example, a segmentation map for each damage type of the vehicle. As in the example illustrated in FIG. 7 , the vehicle damage recognition result 730 may be generated for each vehicle damage type. For example, a recognition result of vehicle damage corresponding to a damage type of cracking, a recognition result of vehicle damage corresponding to a damage type of dent, and a recognition result of vehicle damage corresponding to a damage type of paint peeling It can be created by the first segmentation module 430 .

In an embodiment, the second segmentation module 430 may output a vehicle part recognition result 740 based on the output of the feature extraction module 420 and the output of the vehicle recognition module 500 . Additionally, the second segmentation module 440 may output a vehicle part recognition result 740 based on the vehicle image 710 additionally. In one embodiment, as shown in FIG. 7 in the vehicle part recognition result 740, parts in the vehicle (front bumper, fog lights, fenders, headlights and/or side steps, etc.) may be displayed to be distinguished from each other. .

In an embodiment, the post-processing module 600 may output a vehicle damage recognition result 750 for each vehicle part based at least in part on the vehicle damage recognition result 730 and the vehicle part recognition result 740. there is. Additionally, the post-processing module 600 may generate a recognition result 750 based additionally on the vehicle image 710 . For example, the recognition result 750 includes a result in which a specific part and a specific damage type are mapped, such as “the front passenger seat fender is broken”, “the front bumper is dented”, or “the paint on the rear bumper is peeled off”. can do. As a further example, the recognition result 750 may include a result of damage occurring across two or more parts, such as “dented across the front and rear bumpers”.

In the technique according to an embodiment of the present disclosure, by sharing the parameters of the feature extraction module 420, efficiency of the entire operation can be achieved and computing resources can be efficiently utilized. In the technique according to an embodiment of the present disclosure, since semantic segmentation information for recognizing vehicle damage and instance segmentation information for recognizing vehicle parts can be generated by sharing the feature extraction module 420, the same quantity of Even more accuracy can be achieved by training an artificial neural network using a training dataset.

In the technique according to an embodiment of the present disclosure, a single feature extraction module 420 is shared for the first segmentation module 430 and the second segmentation module 440 without generating two different feature extraction modules. Therefore, not only can the memory usage required in the computing device be reduced during the learning process and the inference process, but also the technical effect of reducing the computation speed can be achieved by reducing the computation required in the learning process and the inference process by half.

In the technique according to an embodiment of the present disclosure, when there is no vehicle by using the vehicle recognition module 500, the execution of the following module(s) is stopped and the user is requested to take a picture again, thereby reducing the delay time and the amount of calculation required. there is. In addition, the technique according to an embodiment of the present disclosure may automatically select a representative vehicle based on the area of the vehicle in a situation where two or more vehicles exist in the vehicle image. Therefore, even if the user does not perform a separate vehicle selection, it is possible to efficiently identify the part and damage of the intended target vehicle.

In a technique according to an embodiment of the present disclosure, through a post-processing module 600 that efficiently combines a semantic segmentation map for vehicle damage and an instance segmentation map for vehicle parts, damage to a vehicle is detected across a plurality of vehicle parts. Even when damage occurs, since damage information is derived on a pixel-by-pixel basis and then combined with vehicle part information, damage spanning multiple parts can be accurately classified for each vehicle part. In a technique according to an embodiment of the present disclosure, in separating damage information spanning a plurality of vehicle parts by part, a vehicle part projection module 610 and a damage segmentation module 620 for each part are used to divide damaged areas. The amount of computation can be greatly reduced.

10 illustrates an exemplary method for providing vehicle parts information according to one embodiment of the present disclosure.

The method of FIG. 10 may be performed by, for example, the computing device 100 of the present disclosure. The order of the steps illustrated in FIG. 10 is expressed as an example for convenience of explanation, and as an example, steps 1010 and 1020 may be performed in parallel with each other.

It is very difficult for general users to inquire about the price of the damaged part when their vehicle is damaged. The reason for this is, firstly, that it is difficult for general users to know in what form each part of the vehicle is divided, and secondly, even if they can classify the parts of the vehicle, they do not know the code information for the necessary parts, so they inquire the price of parts for repair or replacement. that is difficult to do.

According to an embodiment of the present disclosure, a user may easily obtain desired vehicle part information (eg, price information, inventory information, delivery period information, etc.) by photographing a vehicle and uploading the photographed image. there is.

As shown in FIG. 10 , the computing device 100 may identify a vehicle part in a vehicle image by using a pretrained vehicle part recognition module that takes the vehicle image as an input ( 1010 ).

In one embodiment of the present disclosure, a vehicle image may refer to any type of image including a vehicle photographed by a user. An image of a damaged or damaged vehicle or an image including a plurality of vehicles may all be included in the scope of the vehicle image in the present disclosure.

In a further embodiment of the present disclosure, the computing device 100 selects a representative vehicle by filtering vehicles less than a predetermined threshold area in an image input by the user and determines a boundary of the representative vehicle in the input image. It is possible to create a vehicle image that includes. Accordingly, the vehicle image generated in this way may be input, and the vehicle part recognition module may perform identification of parts or parts in the vehicle image.

In one embodiment of the present disclosure, the vehicle part recognition module may refer to a segmentation module capable of recognizing a boundary surface for each part of the vehicle. As an example, the vehicle part recognition module may include an instance segmentation module. The vehicle part recognition module may output where a region of a specific part of the vehicle is located and what a part name for the region is. The vehicle part recognition module may correspond to, for example, the second segmentation module 440 in FIG. 4 .

As such, the vehicle part recognition module may output information for identifying vehicle parts included in the vehicle image. For example, the vehicle part recognition module may include coordinate information about vehicle parts present in the vehicle image, coordinate information about vehicle boundary surfaces present in the vehicle image, and/or a vehicle part class indicating what the vehicle part is in the vehicle image. Quantitative information about can be output.

In one embodiment, the computing device 100 may generate identification information for parts of the vehicle from the vehicle image ( 1020 ).

Identification information on vehicle parts may include information indicating which parts correspond to the vehicle parts identified in step 1010 . For example, identification information on vehicle parts may include code information or serial number information for identifying vehicle parts. For example, the code information or serial number information may refer to information determined by a manufacturer, for example. For example, the identification information on parts of a vehicle may refer to identification information on parts corresponding to vehicle information.

In one embodiment, identification information on vehicle parts may be obtained based on vehicle identification information obtained by recognizing the vehicle and identification information on vehicle parts obtained by recognizing vehicle parts. For example, information about a vehicle model, vehicle identification number, date of manufacture, etc. may be acquired through vehicle recognition, and information about what parts of the vehicle are may be obtained through vehicle part recognition.

In an embodiment, the computing device 100 may generate price information for each part of the vehicle image based on the vehicle part and the identification information (1030).

In an embodiment, the computing device 100 may determine part information corresponding to a vehicle part and information on the number of parts required for parts replacement, and generate price information for each part based on the part information and the number information of parts. there is. For example, the computing device 100 may provide information on parts to be replaced according to vehicle damage. In this case, the computing device 100 determines the damaged part and type of each part of the vehicle in the vehicle image, and determines which parts are to be replaced in correspondence with the damaged parts and damage types and how many parts are to be replaced. can decide

In a further embodiment, the computing device 100 generates quantitative information about a damage type class representing the damage type of a vehicle in the vehicle image using a pretrained damage type identification module that takes a vehicle image as an input. You may. Here, the damage type identification module may perform multi-label based semantic segmentation outputting a first probability value of a first damage type class and a second probability value of a second damage type class in units of pixels in the vehicle image. In one embodiment, the damage type identification module may correspond to the first segmentation module 430 described above. The damage type identification module may identify the damage type of the vehicle by using the output (eg, semantic feature) generated from the feature extraction module as an input. Accordingly, the damage type of the vehicle and location information of the vehicle damage may be matched, and information on parts to be replaced or repaired for the vehicle may be accurately generated based on the matched damage type for each damage location of the vehicle. there is. A detailed description of the damage type module will be replaced with the description of the first segmentation module 430 described above.

In the past, in order to inquire parts information for each part of a vehicle, there was a limitation that the user had to directly check the part code along the hierarchical structure of the part using an Electronic Part Catalog (EPC) system, etc., but according to an embodiment of the present disclosure, A technical effect of being able to easily check codes and information of major vehicle parts can be achieved only with images taken by the user.

11 illustratively illustrates operations of modules for providing parts information of a vehicle according to an embodiment of the present disclosure. Descriptions of the above-described features among the operations and/or functions of the modules shown in FIG. 11 will be omitted to prevent duplication.

According to an embodiment of the present disclosure, computing device 100 may receive input image 1110 . The input image 1110 may include a vehicle-related image captured by a user.

According to an embodiment of the present disclosure, the vehicle recognition module 1200 may identify a representative vehicle from an input image. In one embodiment, the vehicle recognition module 1200 may determine a representative vehicle within the input image 1110 by filtering vehicles below a predetermined threshold area within the input image 1110 . For example, when there is no recognized vehicle, the vehicle recognition module 1200 may determine a re-photographing attempt. When there are two or more recognized vehicles, the vehicle recognition module 1200 induces a user's manual selection by generating a bounding box including boundary surfaces of each of the vehicles existing in the input image 1110, or Among them, a vehicle occupying the largest area within the input image 1110 may be automatically determined as a representative vehicle.

As described above, the vehicle recognition module 1200 may refer to a module for recognizing or selecting a vehicle intended by a user in the input image 1110 . That is, the vehicle recognition module 1200 may determine a representative vehicle from the vehicle image 1110 . The vehicle recognition module 1200 can ensure the accuracy of the operations of the vehicle part recognition module 1120 and the vehicle information extraction module 1300 by clearly determining a vehicle to be analyzed within the vehicle image 1110. . In one embodiment, the vehicle recognition module 1200 in the present disclosure may correspond to the vehicle recognition module 500 described above.

In one embodiment, the operation, structure and function of the vehicle recognition module 1200 will be described later with reference to FIG. 12 .

In one embodiment of the present disclosure, the vehicle recognition module 1200 is connected to an output of a feature extraction module (not shown) to receive an input image including a semantic feature (semantic feature) and a representative vehicle in the input image. A vehicle image including information of may be generated.

Although not shown in FIG. 11 , the input image 1110 may be passed to a feature extraction module (not shown), and semantic features of the input image 1110 may be extracted from a feature extraction model. Here, the feature extraction module (not shown) may correspond to the feature extraction module 420 described above. In this embodiment, the input image 1110 is passed to the feature extraction module, and semantic features (eg, semantic feature maps) for the input image 1110 generated by the feature extraction module are sent to the vehicle recognition module 1220, the vehicle It may be transmitted to the part recognition module 1120 and/or the vehicle information extraction module 1300. As such, the semantic features generated by the feature extraction module can be shared with other modules, and accordingly, the accuracy of vehicle recognition module 1220 for vehicle recognition and the accuracy of vehicle part recognition module 1120 for vehicle recognition accuracy. And/or accuracy of extracting vehicle identification information of the vehicle information extraction module 1300 may be increased.

In one embodiment, the feature extraction module is self-supervised learning before performing a common learning process that is learned together with at least one of the vehicle part recognition module 1120, the vehicle recognition module 1200, or the vehicle information extraction module 1300. It can be independently pre-trained based on the (self-supervised learning) method. Pre-learning of this feature extraction module can be performed together with the decoder module. The feature extraction module generates a first semantic feature from a learning vehicle image in which a part of the first image is masked, and the decoder module restores a second image corresponding to the first image from the first semantic feature, and the The pre-learning may be performed based on a loss function between the first image and the second image. For specific features of the feature extraction module, reference will be made to the details described above with reference to FIGS. 3 and 4 .

In one embodiment of the present disclosure, the vehicle part recognition module 1120 is connected to the vehicle recognition module 1200 to receive an output of the vehicle recognition module 1200 and identify the vehicle part in the vehicle image. can do. The output of the vehicle recognition module 1200 may include, for example, information identifying a representative vehicle in an input image and semantic feature information of the input image.

In one embodiment, the vehicle part recognition module 1120 may perform segmentation in parts of the vehicle. For example, the vehicle part recognition module 1120 may generate a mask map matrix for each part. A position corresponding to a specific part of the vehicle may be displayed as 1 or another specific value on the mask map matrix, and a position not corresponding to a part may be displayed as 0 on the mask map matrix.

In an additional embodiment, the vehicle part recognition module 1120 may generate information to be input to the parts information generation module 1130 by using the output from the vehicle recognition module 1200 and the output from the feature extraction module as inputs.

In one embodiment, the vehicle part recognition module 1120 is an instance segmentation module capable of recognizing a boundary surface of a part of a vehicle in an image, and a region for the part of the vehicle and a probability value that the region corresponds to the part of the vehicle. can output The vehicle part recognition module 1120 may output coordinate information about parts of the vehicle and quantitative information about the vehicle part class indicating which part of the vehicle is in the vehicle image.

In one embodiment, the vehicle part recognition module 1120 may correspond to the previously described second segmentation module 440 . Therefore, the characteristics of the vehicle part recognition module 1120 will be replaced with descriptions related to the structure, operation, and function of the second segmentation module 440 described above. The vehicle part recognition module 1120 and the second segmentation module 440 may be used interchangeably.

In a further embodiment of the present disclosure, although not shown in FIG. 11 , the computing device 100 uses a pre-learned damage type identification module that takes a vehicle image as an input and uses a damage type identification module to indicate the damage type of the vehicle in the vehicle image. It is also possible to generate quantitative information about type classes. Here, the damage type identification module may identify a region and/or type of damage or damage to the vehicle that has occurred within the vehicle image. For example, the damage type identification module may perform multi-label based semantic segmentation outputting a first probability value of a first damage type class and a second probability value of a second damage type class in units of pixels in a vehicle image. .

In one embodiment, the damage type identification module may be used interchangeably with the first segmentation module 430 described above. Therefore, a detailed description of the damage type identification module will be replaced with the above-described features of the first segmentation module 430 .

In one embodiment of the present disclosure, the vehicle information extraction module 1300 may detect a license plate object within the recognized representative vehicle. Detection of such license plate objects may utilize any type of object detection technology. The vehicle information extraction module 1300 may perform vehicle number recognition within a corresponding license plate object through optical character recognition (eg, OCR) technology. The vehicle information extraction module 1300 proceeds to the next step when the vehicle number is normally recognized, and if not, it may generate a notification, pop-up message or message to directly receive the vehicle number from the user.

In one embodiment, the vehicle information extraction module 1300 may recognize a license plate object, determine a license plate number from the license plate object, and obtain part information based on the recognized license plate number.

In one embodiment, the vehicle information extraction module 1300 may query a vehicle identification number DB prepared in advance using the vehicle number as an input, and return the vehicle identification number, which is unique information given when the vehicle is shipped. .

The vehicle information extraction module 1300 will be described in detail with reference to FIG. 13 .

In an embodiment of the present disclosure, the parts information acquisition module 1400 may extract metadata of a vehicle engraved in the vehicle identification number through a decoding operation from the vehicle identification number. The metadata extracted at this time may include vehicle manufacturer, vehicle model, vehicle body type, transmission, engine code, production year, and the like. The part information acquisition module 1400 queries a part information DB obtained in advance based on the extracted metadata and returns part codes constituting major vehicle parts and the number of part codes required for replacement. The part information acquisition module 1400 may obtain part code and quantity information for each vehicle part based on the extracted metadata. For example, radiator grill, front bumper, rear bumper, bonnet, front fender, rear fender, wheel, side mirror (left), side mirror (right), headlight, tail light attached to the trunk side, attached to the quarter panel side Code and quantity information of parts corresponding to the rear light, front door, rear door, trunk, and/or side step may be acquired. The part information acquisition module 1400 may acquire a part code string required for replacement for each vehicle part and a quantity required for replacement for each part code.

In FIG. 11 , the parts information acquisition module 1400 receives the output of the vehicle information extraction module 1300 as an example as an example. According to an implementation aspect, the part information acquisition module 1400 is based at least in part on the output of the vehicle part recognition module 1120 and/or the output of the vehicle information extraction module 1300, in response to the input image 1110, to obtain the required information. Information about parts can be output. As such, depending on the implementation aspect, the parts information acquisition module 1400 may be used interchangeably with the part information generation module 1130 if necessary. For example, one of the part information acquisition module 1400 and part information generation module 1130 may be replaced by the other one.

In an embodiment of the present disclosure, the parts information generation module 1130 may generate result information 1140 based on the output of the vehicle part recognition module 1120 and the output of the part information acquisition module 1400 .

In an embodiment, the parts information generation module 1130 may generate information for each main part of a vehicle searched in a vehicle image input by a user. For example, the parts information generation module 1130 may obtain average area center coordinates of non-zero elements in the mask map matrix for each part generated by the vehicle part recognition module 1120 . That is, the part information generating module 1130 may calculate the coordinates of the center of the area of the corresponding part. The parts information generation module 1130 may match parts information (price, discount price, inventory, expected delivery time, repair cost for the part) for each recognized vehicle part instance. For example, part information may be generated for each specific part. The parts information generation module 1130 may generate result information 1140 by overlapping parts information corresponding to each vehicle part instance on a vehicle image. For example, the part information generation module 1130 overlaps a speech bubble indicating a price for a specific part on a specific part location on a vehicle image, and displays each speech bubble along with a leader line indicating the area center of the corresponding vehicle part. Result information 1140 may be generated.

In general, it may be very difficult for ordinary users to inquire about the price of parts for the damaged area when their vehicle is damaged. Two factors can be cited as the reason for the increase. First, it is difficult for general users to know how each part of the vehicle is divided, and second, even if the vehicle parts can be classified, the corresponding parts required for part price inquiry can be cited. It is difficult to find information about the part code for this product. That is, it is necessary to identify the part code through a professional drawing produced by the manufacturer. In one embodiment of the present disclosure, a part information search unit automatically or manually inputs the image and vehicle number of a vehicle for which a user wants to inquire a price, and automatically inquires a part code and price for each vehicle part, and augmented reality on the vehicle image. A system composed of a display unit displaying a part code and a price for each part of the vehicle may be proposed. Accordingly, as a result, through the present disclosure, a user may be able to inquire about information (price, stock, expected delivery period, parts repair cost, etc.) for each vehicle part only by taking a picture. Through the present disclosure, the user can intuitively check where the damage to the vehicle has occurred and what the damaged part is, just by taking a picture, and furthermore, the user can also grasp information for each part for repair corresponding to the damage to the vehicle. there is.

According to one embodiment of the present disclosure, even if only the vehicle number is input, a technical effect of confirming information on major vehicle parts compatible with the corresponding vehicle can be achieved. According to an embodiment of the present disclosure, parts information is visually displayed on the input vehicle image to users who are not familiar with the classification of vehicle parts, so that the price range and parts information of the part desired to be replaced can be immediately checked.

12 illustratively illustrates a vehicle recognition module 1200 according to an embodiment of the present disclosure.

As shown in FIG. 12 , the vehicle recognition module 1200 may include a vehicle instance filtering module 1210 and a vehicle area comparison module 1220 .

In one embodiment, the vehicle recognition module 1200 may identify a vehicle to be analyzed on a captured image. That is, the vehicle recognition module 1200 may refer to a module that automatically selects a representative vehicle for which a user wishes to inquire parts information when a plurality of vehicles exist in an input vehicle image. For example, the vehicle recognition module 1200 may automatically identify a representative vehicle in a situation where a plurality of vehicles exist on a photographed image or may induce a user to manually select a representative vehicle.

The purpose, function and/or structure of the vehicle recognition module 1200 in FIG. 12 may correspond to the vehicle recognition module 500 in FIGS. 4 and 5 . Accordingly, specific features of the vehicle recognition module 1200 in FIG. 12 and specific features of the vehicle recognition module 500 in FIGS. 4 and 5 can be mutually referenced.

In one embodiment, the vehicle recognition module 1200 may include a vehicle instance filtering module 1210 . As an example, the vehicle instance filtering module 1210 may include an instance segmentation model capable of recognizing a boundary surface of a vehicle along the contour of the vehicle. In the present disclosure, the instance segmentation model may include, for example, Mask R-CNN, Cascade Mask R-CNN, and the like. In an embodiment, the vehicle instance filtering module 1210 may receive a captured image and/or a semantic feature generated by the feature extractor and display a corresponding vehicle area for each vehicle existing in the captured image. For example, a vehicle area displayed for each vehicle may include a mask map matrix composed of 0's and 1's. As an example, the mask map matrix may have a size of [W x H], where W represents the number of horizontal pixels of the mask map and H represents the number of vertical pixels of the mask map. A mask map matrix that displays the area of the vehicle assuming a 2-dimensional image is exemplified, but according to the implementation aspect, when a 3-dimensional image is imported, the mask map matrix also corresponds to the dimension of the image, such as [W x H x L] It will be clear to those skilled in the art that it can be configured in the form.

This vehicle instance filtering module 1210 may correspond to the vehicle instance filtering module 510 in FIG. 5 , and features thereof may be mutually referenced.

In one embodiment of the present disclosure, the vehicle recognition module 1200 may include a vehicle area comparison module 1220 . The vehicle area comparison module 1220 may select a vehicle having the largest area based on a segmentation result of each vehicle output by the vehicle instance filtering module 1210 . For example, the vehicle area comparison module 1220 assumes that among a plurality of recognized vehicles, a vehicle for which the user wants to inquire parts information has the largest area, and the vehicle instance filtering module 1210 outputs each vehicle for each vehicle. It may refer to a module for finding an object having the widest non-zero area from the mask map. In additional embodiments, the operation of vehicle area comparison module 1220 may be replaced by vehicle instance filtering module 1210 .

In one embodiment, the vehicle area comparison module 1220 may correspond to the vehicle selection module 520 in FIG. 5 , and features thereof may also be cross-referenced.

Also, the vehicle recognition module 1200 in FIG. 12 may perform an operation algorithm according to the number of recognized vehicles, as illustrated in FIG. 8 described above. In one embodiment, when the number of vehicles recognized by the vehicle recognition module 1200 is 0, execution of the algorithm may be terminated or information for requesting re-photographing may be generated. In one embodiment, when the number of vehicles recognized by the vehicle recognition module 1200 is one, the area comparison algorithm of the vehicle area comparison module 1220 is not performed, and the result of the recognized vehicle is transmitted to the next step. can In one embodiment, when the number of vehicles recognized by the vehicle recognition module 1200 is two or more, an area having the largest area may be selected from among area information obtained for each vehicle. For example, a matrix having the largest number of non-zero elements in the mask map matrix for each vehicle may be selected. Then, the vehicle area comparison module 1220 may generate a representative vehicle image based on the area having the largest area. For example, the vehicle area comparison module 1220 multiplies the mask map matrix corresponding to the selected representative vehicle and the matrix of the input vehicle image element by element, leaving a portion corresponding to the representative vehicle area and replacing the remaining portion with 0. A vehicle image can be created.

13 illustratively illustrates a vehicle information extraction module 1300 according to an embodiment of the present disclosure.

As shown in FIG. 13 , the vehicle information extraction module 1300 may include a license plate object recognition module 1310 and a vehicle identification number extraction module 1320 .

In one embodiment, the license plate object recognition module 1310 of the vehicle information extraction module 1300 may detect a license plate object existing in front or behind the recognized representative vehicle. For example, the vehicle information extraction module 1300 may include a pretrained segmentation module or object detection module to detect license plates in a vehicle. For example, the license plate object recognition module 1310 uses an object detection algorithm for a matrix corresponding to an image of a representative vehicle to generate a bounding box of a minimum size including a license plate, and a confidence score for the generated bounding box. can be printed out. The license plate object recognition module 1310 may automatically select one license plate object based on the reliability score when two or more license plate objects are detected. In another embodiment, if two or more license plate objects are detected, the license plate object recognition module 1310 may present the detected license plate objects to the user and determine a specific license plate object in response to user selection. In one embodiment, the license plate object recognition module 1310 may generate request information for allowing the user to directly input the license plate number when the detected license plate object does not exist.

In one embodiment, the license plate object recognition module 1310 of the vehicle information extraction module 1300 may be pre-learned based on training data in which an area for a license plate is labeled in a vehicle image. As another example, the license plate object recognition module 1310 of the vehicle information extraction module 1300 may be learned based at least in part on the semantic features of the vehicle image generated by the feature extraction module. As another example, the license plate object recognition module 1310 of the vehicle information extraction module 1300 may be pretrained based on vehicle images and semantic features.

In one embodiment, the vehicle identification number extraction module 1320 of the vehicle information extraction module 1300 uses location information (eg, coordinate information) corresponding to the license plate object recognized by the license plate object recognition module 1310 to represent the representative vehicle. The license plate object area in the image may be checked and the license plate number for the license plate object area may be recognized through OCR technology. Additionally, the vehicle identification number extraction module 1320 may extract a vehicle identification number corresponding to the recognized vehicle number. Extraction of the vehicle identification number may be performed by accessing the vehicle identification number DB and querying the vehicle identification number DB. Accordingly, the vehicle identification number extraction module 1320 may extract the vehicle identification number, which is unique information given when the vehicle is shipped, using the vehicle number recognized through OCR or the like.

In an embodiment, the vehicle identification number extraction module 1320 verifies the recognized vehicle number, and if the vehicle number has a normal standard, the vehicle identification number and/or vehicle identification number may be transmitted to the part information obtaining module 1400. if. If the recognized vehicle number does not conform to the standard, the vehicle identification number extraction module 1320 may generate request information for allowing the user to directly input the vehicle number.

As described above, since the vehicle information extraction module 1300 automatically detects vehicle identification information within a photographed image, vehicle identification information, which is a prerequisite for obtaining part information, is efficiently obtained without user effort. It can be.

14 illustratively illustrates a component information acquisition module 1400 according to an embodiment of the present disclosure.

As shown in FIG. 14, the parts information acquisition module 1400 includes a metadata acquisition module 1410, a part code lookup module 1420, a part price information lookup module 1430, and a part price information generation module 1440. can include

In one embodiment, the part information acquisition module 1400 in FIG. 14 may perform the operation of the part information generation module 1130 in FIG. 11 . In this case, the part information acquisition module 1400 in FIG. 11 may operate based on the result of the vehicle part recognition module 1120 and/or the result of the vehicle information extraction module 1300 . In a further embodiment, the part information acquisition module 1400 may be operated further based on damage type and/or damage area information of the vehicle.

In an embodiment, the parts information obtaining module 1400 may obtain and generate information related to vehicle parts corresponding to a vehicle number or vehicle identification number.

In an embodiment, the metadata acquisition module 1410 may extract vehicle metadata included in the vehicle identification number through a decoding operation from the input vehicle identification number or vehicle number. In one embodiment, meta data may include various types of identification information assigned to a vehicle. For example, the meta data may include vehicle manufacturer, vehicle type, vehicle body type, transmission, engine code, and/or production year.

In one embodiment, the part code inquiry module 1420 queries the part information DB based on the meta data, and as a result of the search for the part information DB, part code information related to the vehicle and information on the number of part codes required for replacement. can return The part code inquiry module 1420 may obtain part code and quantity information for each part of the vehicle. Parts of a vehicle in the present disclosure include, for example, a radiator grill, a front bumper, a rear bumper, a bonnet, a front fender, a rear fender, a wheel, a side mirror, a headlight, a tail light, a front door, a rear door, a trunk, and a side step. etc. may be included. The part code inquiry module 1420 may return a character string for identifying a part code required for replacement for each part of the vehicle and information on the number thereof. For example, codes and numbers of parts corresponding to “radiator grill” may be exemplified as follows.

Radiator Grille -> "2058880023": 1, "20588802609982": 1, "0008880060": 2, "0008171016": 13

As in the above example, code numbers of four parts related to the radiator grill and number information of each part may be output by the part code inquiry module 1420 .

In an additional embodiment, the part code inquiry module 1420 may return part code and quantity information corresponding to parts related to damage to the vehicle, among parts code and quantity information for each part of the vehicle. In addition, the part code inquiry module 1420 may return part codes and quantity information corresponding to parts requiring repair among part codes and quantity information for each part of the vehicle. Parts related to damage and/or parts requiring repair may be determined by the vehicle damage recognition module and the vehicle part recognition module.

In an embodiment, the parts price information inquiry module 1430 may search a parts information DB prepared in advance based on part codes by inputting part codes and quantity information for each vehicle part. The part price information inquiry module 1430 may return not only the price of the part corresponding to the part code, but also information on the discount price, inventory, and/or expected delivery period of the part.

For example, in the example of the radiator grill parts described above, the part price information inquiry module 1430 may return the following information for a specific part code.

"2058880023": { "list_price": 53000, "promotion_price": 50000, "left_inventory": 3, "estimate_delivery_day": 13 }

In an embodiment, the parts price information generation module 1440 may generate information for each major part of a vehicle inquired into a vehicle image input by a user. The part price information generation module 1440 may generate any type of additional information related to a specific part, such as the above-mentioned discount price, inventory, and estimated delivery period, as well as the price of the part.

In one embodiment, the parts price information generation module 1440 may be used interchangeably with the parts information generation module 1130 in FIG. 11 .

In an embodiment, the part price information generation module 1440 may calculate average area center coordinates of non-zero elements for each mask map matrix for each part output from the vehicle part recognition module 1120 . The part price information generation module 1440 may calculate center coordinates of the area of a specific part.

In an embodiment, the part price information generation module 1440 may match part information (eg, price, discount price, inventory, estimated delivery time, etc.) corresponding to each recognized vehicle part.

In one embodiment, the parts price information generation module 1440 may generate a UI of any form containing corresponding part information for each vehicle part. For example, the part price information generation module 1440 may generate UI information by overlapping a speech bubble containing part information corresponding to each vehicle part on a vehicle image.

In an embodiment, the part price information generation module 1440 may generate a resultant image in which price information for the part is added in augmented reality form to the average coordinates of the center of an area in a photographed image captured by a user. The resulting image added in the form of augmented reality may vary in its output form or whether or not it is output according to interaction with the user.

15 illustratively illustrates information 1500 for each part of the vehicle generated by the vehicle part recognition module 1120 according to an embodiment of the present disclosure.

In an embodiment, the vehicle part-specific information 1500 may include segmentation result information 1510b, 1520b, 1530b, and 1540b indicating boundary lines of the vehicle parts.

In an embodiment, the vehicle part-specific information 1500 may include identification information about the vehicle part and result information 1510a, 1520a, 1530a, and 1540a indicating a bounding box thereof.

As described above, the vehicle part recognition module 1120 may classify each of the parts of the vehicle, and information on at least one part corresponding to each of the parts classified in this way is inquired, and replacement or repair is required. As additional information (eg, price information) on the part is output, the user can intuitively recognize information on the part to be replaced. Furthermore, based on the damage information for each part of the vehicle, additional information (eg, price information, etc.) on at least one part corresponding to the part requiring repair may be generated, so that the user can be When a photographed image is uploaded, information about the accident site, damage type, and parts to be replaced to solve the damage can be intuitively displayed.

16 illustratively illustrates vehicle parts information 1600 generated according to an embodiment of the present disclosure.

16 illustrates an example in which vehicle parts information 1600 is output in the form of augmented reality on a photographed image. Price information corresponding to parts of a vehicle shown in the form of augmented reality is an example, and other information in any form related to parts as well as price information may also be included in the present disclosure.

In one embodiment, the user may interact with part information shown in augmented reality, and output of a part of the part information may be varied according to the interaction.

As shown in FIG. 16, as price information corresponding to a specific part or part of a vehicle is intuitively displayed, the user can intuitively check the accident part, damage type, and part information to be replaced to solve the damage. .

In an embodiment, the computing device 100 may display part information for each recognized vehicle part in the form of a table or list to solve this problem, since there is room for excessive overlapping of speech bubbles when there are many recognized parts of the vehicle.

In an additional embodiment, the computing device 100 may display only necessary information for the user by focusing on part information corresponding to the damaged vehicle part or outputting only the corresponding part information.

According to embodiments of the present disclosure, since part information is visually displayed on an input vehicle image to users who are not familiar with the classification of vehicle parts, it is possible to intuitively provide price range and parts information of a part desired to be replaced.

In one embodiment of the present disclosure, when damage to a vehicle occurs, a photographed image including an area where the damage occurs can be analyzed to show not only information about the damage but also information on parts to be replaced to solve the damage. Therefore, user convenience can be maximized.

17 depicts a simplified and general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

A component, module or unit in this disclosure includes routines, procedures, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In addition, those skilled in the art will understand that the methods presented in this disclosure can be used in single-processor or multiprocessor computing devices, minicomputers, mainframe computers as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like ( It will be fully appreciated that each of these may be implemented with other computer system configurations, including those that may be operative in connection with one or more associated devices.

Embodiments described in this disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

A computing device typically includes a variety of computer readable media. Computer readable media can be any medium that can be accessed by a computer, including volatile and nonvolatile media, transitory and non-transitory media, removable and non-transitory media. Includes removable media. By way of example, and not limitation, computer readable media may include computer readable storage media and computer readable transmission media.

Computer readable storage media are volatile and nonvolatile media, transitory and non-transitory, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. includes media Computer readable storage media may include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device. device, or any other medium that can be accessed by a computer and used to store desired information.

A computer readable transmission medium typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism. Including all information delivery media. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information within the signal. By way of example, and not limitation, computer readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer readable transmission media.

An exemplary environment 2000 implementing various aspects of the present invention is shown comprising a computer 2002, which includes a processing unit 2004, a system memory 2006 and a system bus 2008. do. Computer 200 herein may be used interchangeably with a computing device. System bus 2008 couples system components, including but not limited to system memory 2006, to processing unit 2004. Processing unit 2004 may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be used as the processing unit 2004.

System bus 2008 may be any of several types of bus structures that may additionally be interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 2006 includes read only memory (ROM) 2010 and random access memory (RAM) 2012 . The basic input/output system (BIOS) is stored in non-volatile memory 2010, such as ROM, EPROM, EEPROM, etc. BIOS is a basic set of information that helps transfer information between components within the computer 2002, such as during startup. contains routines. RAM 2012 may also include high-speed RAM, such as static RAM, for caching data.

The computer 2002 may also read from an internal hard disk drive (HDD) 2014 (eg EIDE, SATA), a magnetic floppy disk drive (FDD) 2016 (eg a removable diskette 2018), or for writing to them), SSDs and optical disk drives 2020 (for example, for reading CD-ROM disks 2022 or reading from or writing to other high capacity optical media such as DVDs) include The hard disk drive 2014, magnetic disk drive 2016, and optical disk drive 2020 are connected to the system bus 2008 by the hard disk drive interface 2024, magnetic disk drive interface 2026, and optical drive interface 2028, respectively. ) can be connected to The interface 2024 for external drive implementation includes, for example, at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer readable media provide non-volatile storage of data, data structures, computer executable instructions, and the like. In the case of computer 2002, drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable storage media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art can use zip drives, magnetic cassettes, flash memory cards, cartridges, It will be appreciated that other types of computer-readable storage media, such as those of other types, may also be used in the exemplary operating environment and that any such media may contain computer-executable instructions for performing the methods of the present invention. .

A number of program modules may be stored on the drive and RAM 2012, including an operating system 2030, one or more application programs 2032, other program modules 2034, and program data 2036. All or portions of the operating system, applications, modules and/or data may also be cached in RAM 2012. It will be appreciated that the present invention may be implemented in a variety of commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer 2002 through one or more wired/wireless input devices, such as a keyboard 2038 and a pointing device such as a mouse 2040. Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. Although these and other input devices are often connected to the processing unit 2004 through an input device interface 2042 that is connected to the system bus 2008, a parallel port, IEEE 1394 serial port, game port, USB port, IR interface, may be connected by other interfaces such as the like.

A monitor 2044 or other type of display device is also connected to the system bus 2008 through an interface such as a video adapter 2046. In addition to the monitor 2044, computers typically include other peripheral output devices (not shown) such as speakers, printers, and the like.

Computer 2002 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 2048 via wired and/or wireless communications. Remote computer(s) 2048 may be workstations, server computers, routers, personal computers, handheld computers, microprocessor-based entertainment devices, peer devices, or other common network nodes, and generally relate to computer 2002. Although many or all of the described components are included, for brevity, only memory storage device 2050 is shown. The logical connections shown include wired/wireless connections to a local area network (LAN) 2052 and/or a larger network, such as a wide area network (WAN) 2054 . Such LAN and WAN networking environments are common in offices and corporations and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to worldwide computer networks, such as the Internet.

When used in a LAN networking environment, computer 2002 is connected to local network 2052 through wired and/or wireless communication network interfaces or adapters 2056. Adapter 2056 may facilitate wired or wireless communications to LAN 2052, which also includes a wireless access point installed therein to communicate with wireless adapter 2056. When used in a WAN networking environment, computer 2002 may include a modem 2058, be connected to a communications server on WAN 2054, or other means of establishing communications over WAN 2054, such as over the Internet. have Modem 2058, which may be internal or external and wired or wireless, is connected to system bus 2008 through serial port interface 2042. In a networked environment, program modules described for computer 2002, or portions thereof, may be stored in remote memory/storage device 2050. It will be appreciated that the network connections shown are exemplary and other means of establishing a communication link between computers may be used.

Computer 1602 is any wireless device or entity that is deployed and operating in wireless communication, such as printers, scanners, desktop and/or portable computers, portable data assistants (PDAs), communication satellites, and associated with wireless detectable tags. It operates to communicate with arbitrary equipment or places and telephones. This includes at least Wi-Fi and Bluetooth wireless technologies. Thus, the communication may be a predefined structure as in conventional networks or simply an ad hoc communication between at least two devices.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of exemplary approaches. Based upon design priorities, it is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure. The method claims of this disclosure present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

Claims (17)

A method for providing parts information of a vehicle performed by a computing device comprising at least one processor, comprising:
generating a semantic feature for the captured image by using a pretrained feature extraction module that takes the captured image of the vehicle as an input;
Using a pretrained vehicle recognition module connected to the output of the feature extraction module, determining a representative vehicle among a plurality of vehicles by filtering vehicles having a predetermined threshold area or less in an input image including the semantic feature, and generating a vehicle image including a boundary surface of the representative vehicle in the input image;
generating a mask map matrix for each part identifying a vehicle part in the vehicle image by using a pretrained vehicle part recognition module that takes the vehicle image as an input;
generating identification information on parts of the vehicle from the vehicle image;
generating price information for each part of the vehicle image based on the vehicle part and the identification information;
determining average area center coordinates of non-zero elements in the mask map matrix for each component;
determining part price information corresponding to the average area center coordinate from the price information for each part; and
generating a result image in which price information for the part is added in an augmented reality form to the average area center coordinates on the photographed image;
includes, and
The feature extraction module is an independently pre-trained model based on a self-supervised learning method before performing a common learning process that is learned together with at least one of the vehicle part recognition module and the vehicle recognition module. Corresponds to, pre-learning of the feature extraction module is performed together with a decoder module, the feature extraction module generates a first semantic feature from a learning vehicle image in which a part of the first image is masked, and the decoder module generates a first semantic feature. Restoring a second image corresponding to the first image from 1 semantic feature, and performing the pre-learning based on a loss function between the first image and the second image.
method. According to claim 1,
The vehicle part recognition module,
An instance segmentation module capable of recognizing a boundary surface of a part of a vehicle, and outputting a region for the part of the vehicle and a probability value that the region corresponds to the part of the vehicle,
method. According to claim 1,
The vehicle part recognition module,
Outputting coordinate information about the part of the vehicle and quantitative information about a vehicle part class indicating what part of the vehicle is in the vehicle image.
method. delete delete According to claim 1,
Determining the representative vehicle,
determining whether to attempt rephotographing when there is no vehicle recognized by the vehicle recognizing module; and
When there are two or more vehicles recognized by the vehicle recognition module, a user's manual selection is induced by creating a bounding box including the boundary surfaces of each of the vehicles existing in the input image, or the input image among the vehicles. Automatically determining a vehicle with the largest area occupied within the vehicle as a representative vehicle
including,
method. delete delete delete According to claim 1,
generating quantitative information about a damage type class representing a damage type of a vehicle in the vehicle image using a pre-learned damage type identification module that takes the vehicle image as an input;
Including more,
method. According to claim 10,
The damage type identification module,
Performing multi-label based semantic segmentation for outputting a first probability value of a first damage type class and a second probability value of a second damage type class in units of pixels in the vehicle image,
method. According to claim 11,
The step of generating price information for each part of the vehicle image,
generating price information for each part corresponding to a damaged part of the vehicle image based on the vehicle part, the identification information, and the damage type;
including,
method. According to claim 1,
The step of generating price information for each part of the vehicle image,
Recognizing a license plate object included in the identification information and obtaining a vehicle number from the license plate object;
extracting a vehicle identification number corresponding to the vehicle number; and
obtaining metadata corresponding to the vehicle identification number by decoding the vehicle identification number;
includes, and
The meta data includes at least one of manufacturer information, vehicle type information, vehicle body type information, transmission information, engine information, or production year information.
method. According to claim 13,
The step of generating price information for each part of the vehicle image,
based on the meta data, generating part information corresponding to a part of the vehicle and information on the number of parts required for parts replacement;
generating price information for each part based on the parts information and the number information of the parts;
Including more,
method. delete A computer program stored on a computer readable storage medium, the computer program, when executed by one or more processors, performing a method for providing parts information of a vehicle, the method comprising:
generating a semantic feature for the captured image by using a pretrained feature extraction module that takes the captured image of the vehicle as an input;
Using a pretrained vehicle recognition module connected to the output of the feature extraction module, determining a representative vehicle among a plurality of vehicles by filtering vehicles having a predetermined threshold area or less in an input image including the semantic feature, and generating a vehicle image including a boundary surface of the representative vehicle in the input image;
generating a mask map matrix for each part identifying a vehicle part in the vehicle image by using a pretrained vehicle part recognition module that takes the vehicle image as an input;
generating identification information on parts of the vehicle from the vehicle image;
generating price information for each part of the vehicle image based on the vehicle part and the identification information;
determining average area center coordinates of non-zero elements in the mask map matrix for each component;
determining part price information corresponding to the average area center coordinate from the price information for each part; and
generating a result image in which price information for the part is added in an augmented reality form to the average area center coordinates on the photographed image;
includes, and
The feature extraction module is an independently pre-trained model based on a self-supervised learning method before performing a common learning process that is learned together with at least one of the vehicle part recognition module and the vehicle recognition module. Corresponds to, pre-learning of the feature extraction module is performed together with a decoder module, the feature extraction module generates a first semantic feature from a learning vehicle image in which a part of the first image is masked, and the decoder module generates a first semantic feature. Restoring a second image corresponding to the first image from 1 semantic feature, and performing the pre-learning based on a loss function between the first image and the second image,
A computer program stored on a computer readable storage medium.
A computing device for providing parts information of a vehicle, the computing device comprising:
It includes one or more processors, the processors:
generating a semantic feature for the captured image by using a pretrained feature extraction module that takes the captured image of the vehicle as an input;
Using a pretrained vehicle recognition module connected to the output of the feature extraction module, determining a representative vehicle among a plurality of vehicles by filtering vehicles having a predetermined threshold area or less in an input image including the semantic feature, and generating a vehicle image including a boundary surface of the representative vehicle in the input image;
generating a mask map matrix for each part identifying a vehicle part in the vehicle image by using a pretrained vehicle part recognition module that takes the vehicle image as an input;
generating identification information for parts of the vehicle from the vehicle image;
generating price information for each part of the vehicle image based on the vehicle part and the identification information;
determining average area center coordinates of non-zero elements in the mask map matrix for each component;
determining part price information corresponding to the average area center coordinate from the price information for each part; and
generating a result image in which price information for the part is added in an augmented reality form to the average area center coordinates on the photographed image;
And
The feature extraction module is an independently pretrained model based on a self-supervised learning method before performing a common learning process that is learned together with at least one of the vehicle part recognition module and the vehicle recognition module. Corresponds to, pre-learning of the feature extraction module is performed together with a decoder module, the feature extraction module generates a first semantic feature from a learning vehicle image in which a part of the first image is masked, and the decoder module generates a first semantic feature. Restoring a second image corresponding to the first image from 1 semantic feature, and performing the pre-learning based on a loss function between the first image and the second image.
computing device.

Images