KR20230112765A - Method, device and program for vehicle AS simulation based on digital twin and vehicle status notification - Google Patents

Abstract

The present invention relates to a digital twin-based vehicle AS simulation and vehicle status notification method, which not only enables accurate and efficient work by AS personnel, but also provides a transparent and reliable AS by providing an AS process to the vehicle owner (customer). can proceed.

Description

Vehicle AS simulation based on digital twin, vehicle status notification method, device and program {Method, device and program for vehicle AS simulation based on digital twin and vehicle status notification}

The present disclosure relates to vehicle AS simulation and a vehicle status notification method.

Recently, technologies in various fields, such as image analysis technology and technology for learning artificial intelligence models, are rapidly developing.

Despite the development of these technologies, most of the tasks such as vehicle maintenance, inspection, and after-sales service are still performed directly by people, and due to this, there are considerable differences in skills and costs among individuals. there is.

Therefore, if this technology is used for vehicle image analysis and AS information derivation using image analysis, it is expected that it can be used in areas such as vehicle maintenance, inspection, and AS, but this technology has not been disclosed at present.

In addition, when the Digital Twin technology is applied, it is expected that it will be able to simulate the vehicle AS and provide notifications on the vehicle status in a realistic way.

The present invention for solving the above problems is to proceed with a virtual AS simulation by inputting an image of a vehicle to be simulated into an artificial intelligence model.

In addition, the present invention is intended to derive AS information for a simulation target vehicle based on virtual AS simulation results, visualize it based on a digital twin, and provide it to an AS person in charge.

In addition, the present invention is intended to provide the owner of the vehicle with AS status information of the vehicle proceeding from the AS person in charge.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

A vehicle AS simulation method based on digital twin according to an embodiment of the present invention for solving the above problems is a method performed by a vehicle AS simulation device, comprising the steps of receiving at least one image of a simulation target vehicle; Analyzing the received image and performing AS simulation on the simulation target vehicle; Deriving AS information on the simulation target vehicle based on the AS simulation result; and generating and outputting visualization information based on the derived AS information.

In addition, the vehicle AS simulation device is characterized in that it outputs the generated visualization information based on the digital twin.

The method may further include providing the owner of the simulation target vehicle with the AS status according to the AS progress for the simulation target vehicle.

In addition to this, another method for implementing the present invention, another system, and a computer readable recording medium recording a computer program for executing the method may be further provided.

According to the present invention as described above, a virtual AS simulation may be performed by inputting an image of a simulation target vehicle to an artificial intelligence model.

In addition, according to the present invention, AS information on the simulation target vehicle can be derived based on the virtual AS simulation result, visualized based on the digital twin, and provided to the AS person in charge.

In addition, according to the present invention, it is possible to provide the owner of the vehicle with AS status information of the vehicle proceeding from the AS person in charge.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a block diagram of an apparatus for deriving analysis information through vehicle image analysis and providing vehicle AS information using the same according to an embodiment of the present invention.
2 is a diagram illustrating an example of generating a learning dataset by analyzing an image of a learning target vehicle and learning an artificial intelligence model using the same.
3 is a diagram illustrating the derivation of AS information by inputting an image of a vehicle to be inspected into an artificial intelligence model.
4 is a diagram illustrating that a virtual AS simulation is performed by inputting an image of a vehicle to be simulated into an artificial intelligence model.
5 is a diagram illustrating that the derived AS information is output on the basis of a digital twin, provided to an AS person in charge, and AS status information is provided to a vehicle owner.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an apparatus for deriving analysis information through vehicle image analysis and providing vehicle AS information using the same according to an embodiment of the present invention.

2 is a diagram illustrating an example of generating a learning dataset by analyzing an image of a learning target vehicle and learning an artificial intelligence model using the same.

3 is a diagram illustrating the derivation of AS information by inputting an image of a vehicle to be inspected into an artificial intelligence model.

In order to explain the digital twin-based vehicle AS simulation, vehicle status notification method, and building an artificial intelligence model for the device according to an embodiment of the present invention, analysis through vehicle image analysis with reference to FIGS. 1 to 3 Information derivation and a vehicle AS information providing device and method using the same will be described first.

Referring to FIG. 1 , an apparatus for deriving analysis information through vehicle image analysis and providing vehicle AS information using the same according to an embodiment of the present invention includes a processor, a communication unit, a database, and an input/output unit.

However, in some embodiments, the device may include fewer or more components than those shown in FIG. 1 .

Depending on the embodiment, it may be divided into an analysis information derivation device through vehicle image analysis and a vehicle AS information providing device, and these specific details are easily selected by the practitioner of the present invention.

In addition, in an embodiment of the present invention, an apparatus for deriving analysis information through vehicle image analysis and an apparatus for providing vehicle AS information may include a computer and information processing means, and may include a server device.

The communication unit communicates with an external device, server, etc., may receive at least one image of the vehicle to be studied or the vehicle to be inspected, and transmit AS information derived about the vehicle to be inspected to the external device or server.

The memory stores various commands, algorithms, and artificial intelligence models for executing a method of deriving analysis information through vehicle image analysis and a method of providing vehicle AS information.

In addition, the memory stores various data sources for vehicle inspection, such as a list of various vehicle types and basic specification information for each vehicle.

The input/output unit has a general input unit and output unit configuration, and can receive various control signals, information, images, etc., and output various generated AS information.

The processor is responsible for controlling the components in the analysis information derivation device and the vehicle AS information providing device through vehicle image analysis, and can execute the method according to the embodiment of the present invention by executing commands and algorithms in the memory and using the artificial intelligence model. there is.

Below, a device for deriving analysis information through vehicle image analysis will be referred to as a first device, and a vehicle AS information providing device will be referred to as a second device.

Referring to FIG. 2 , a method of deriving analysis information through vehicle image analysis according to an embodiment of the present invention will be described.

A first device receives at least one image (video, image) of a learning target vehicle.

The first device analyzes the received image to recognize images of vehicle parts included in the learning target vehicle, locations of the vehicle components in the vehicle, and connection relationships between a plurality of vehicle components.

The first device inputs the result recognized by the artificial intelligence model as learning data together with the vehicle type and basic vehicle specifications.

At this time, it is preferable to receive a plurality of different images for more detailed and accurate analysis.

In an embodiment of the present invention, an image may be a video image composed of a plurality of frames and having a predetermined time, or an image composed of one frame may be applied.

In addition, when individual images of all detachable parts such as the bonnet, headlight, rearview mirror, wheel, body, and trunk are received as well as the image of the exterior of the vehicle, each image may be separated and stored in the memory.

In addition, when an image or coupling information coupled between parts/parts is received, a connection relationship between a plurality of vehicle parts may be recognized based on this, generated as learning data, and stored in a memory.

By recognizing this information, storing it in memory, building big data, and inputting it as learning data to learn an artificial intelligence model, when an image of an accident vehicle or a vehicle in which a breakdown occurs is received, the image is analyzed (big data, ML). Classification by AS can be done only with this, and it is possible to determine the parts that require AS.

As described above, the processor recognizes the location of vehicle parts in the vehicle and the connection relationship between multiple vehicle parts, creates big data with the vehicle type and basic vehicle specifications as a learning data set, and inputs it into an artificial intelligence model for learning will make

In some embodiments, the processor may build and use individual AI models for each vehicle manufacturing brand.

In some embodiments, the processor may build and use individual artificial intelligence models for each type of vehicle.

In this way, if an individual AI model is built and used for each vehicle type, the AI model can start analysis by entering the type of vehicle to be inspected and inputting only the image of the vehicle to be inspected, so efficient, fast and accurate judgment can be made. be able to get off

In some embodiments, the vehicle type may be recognized/identified by matching with the customer's DB.

As an embodiment, the memory may include a vehicle type analysis algorithm, and the processor uses the vehicle type algorithm to determine various types of vehicle body shape, side mirror shape, lighting shape, emblem shape, body size, ratio, color, wheel shape, etc. of the vehicle. It can identify elements accurately and in detail, determine the type of vehicle, and even determine the vehicle's various options.

Referring to FIG. 3 , a method for providing vehicle AS information according to an embodiment of the present invention will be described.

A second device receives at least one image of the vehicle to be inspected.

The second device inspects the vehicle to be inspected by inputting at least one image of the vehicle to be inspected into the artificial intelligence model.

In this case, the second device may also receive input of the type of vehicle to be inspected, basic specifications, and the like.

However, the processor is not limited thereto, and when at least one image of the vehicle to be inspected is received, the artificial intelligence model analyzes the image to determine the type of vehicle, and stores various basic specification information for inspection of the vehicle in memory. can be loaded.

The artificial intelligence model can inspect the vehicle to be inspected using the learned model, and can utilize big data stored in the device/server.

Based on the inspection result, it is determined whether the vehicle to be inspected needs AS.

If the vehicle to be inspected requires AS, at least one AS-required part is derived.

Since the vehicle to be inspected may not necessarily require AS, the processor may determine whether the vehicle to be inspected requires AS by performing the above process.

In addition, when it is determined that the vehicle to be inspected requires AS, the processor may derive which parts require AS.

In addition, the processor may derive an essential AS part in which the AS is essential and a recommended AS part in which the AS is recommended.

In addition, the processor may derive a speculative AS portion that guesses that an AS will be needed in the next few days or months.

At this time, the processor inputs the image of the vehicle to be inspected to the artificial intelligence model, and the artificial intelligence model determines the external shape of the vehicle based on the input image, and determines the part requiring AS through the external shape.

In some embodiments, there may be parts that are difficult to identify only with the external image of the vehicle to be inspected. In this case, at least one survey on the state of the vehicle may be conducted with the user terminal, and based on the survey result, the user may select a vehicle part to be photographed and request image upload.

Based on the learned results, the artificial intelligence model can analyze the video of the vehicle to be inspected to determine various information, as well as perform classification decisions for each AS based on big data. (Example: A car that was brought in with a certain exterior condition had many tire changes, etc.)

AS information for AS of the derived AS-required part is derived.

In some embodiments, the processor may evaluate the reliability of the AS information using an artificial intelligence model, and if a value equal to or less than a preset reliability is calculated, it may request more images around the suspicious part in the vehicle, and the corresponding part A confirmation request signal may be sent to an expert on

In addition, the AS information may include part information required for the AS of the corresponding part, but is not limited thereto since parts are not required and AS is needed in some cases.

Based on the derived AS information, the AS person in charge of the vehicle to be inspected is assigned, and a work movement route is derived.

The memory may store information on career information, fields of expertise, and the like for a plurality of AS personnel.

The processor may match the derived AS information to the contact person list and match the person in charge/expert optimized for the derived AS information.

As an embodiment, in the case of AS for a specific part, an essential responsibility condition that an AS person in charge with a specific qualification must perform may be set.

The processor not only derives the assignment of AS personnel and the work movement path, but also calculates the waiting time for AS and the time required for AS.

Accordingly, the AS information derived by the processor may include various information related to the AS, such as a location where the AS is required, parts required for the AS, an AS schedule, an AS person in charge, and a work movement route for the AS.

In addition, the processor may request various settings for the AS so that the AS person in charge can start the AS after moving to an AS execution location (eg, a workbench, a workbench).

Accordingly, the processor may generate ML-based data through vehicle task allocation and workbay location annotation, and may create a learning/test model.

In addition, the apparatus according to the embodiment of the present invention may use an algorithm for assigning an AS person in charge and a work movement path, and may use a graph algorithm through assigning weights to a person in charge and a workbay based on the analyzed data.

Additionally, the processor may perform labor analysis.

Specifically, the processor may output AS part and labor analysis information through a vehicle image. (Using Hadoop-based big data analysis technology)

4 is a diagram illustrating that a virtual AS simulation is performed by inputting an image of a vehicle to be simulated into an artificial intelligence model.

5 is a diagram illustrating that the derived AS information is output on the basis of a digital twin, provided to an AS person in charge, and AS status information is provided to a vehicle owner.

The processor receives at least one image of the simulation target vehicle through the communication unit.

A processor analyzes the received image and proceeds with a virtual AS simulation of the simulation target vehicle.

The processor derives simulation data for the simulation target vehicle based on the AS simulation result.

A processor generates and outputs visualization information based on the derived AS information.

The processor performs a virtual AS simulation using the artificial intelligence model, and at this time, the artificial intelligence model can first determine whether the simulation target vehicle needs AS based on the image of the simulation target vehicle.

And, the artificial intelligence model derives AS information and uses it for virtual AS simulation. At this time, the AS information includes the location where AS is needed, the parts needed for AS, the AS schedule, the person in charge of AS, the work movement route for AS, etc. Various information related to may be included.

The processor performs a virtual AS simulation for the simulation target vehicle based on AS information using an artificial intelligence model.

And, the processor may derive simulation data based on the simulation result.

At this time, the digital twin-based vehicle AS simulation device simultaneously proceeds with virtual AS simulation while the AS person in charge performs AS, and outputs and provides simulation data in a digital twin method, so that the AS person in charge can refer to the virtual AS simulation video and AS can proceed.

Through this configuration, the processor enables the AS person in charge to systematically and accurately proceed with the AS step by step, and 3D models the customer's vehicle to output status information through AR/VR.

Also, the processor can visualize the AS part through reverse engineering.

Through the above configurations, the vehicle AS simulation device based on the digital twin according to an embodiment of the present invention can guide the AS person in charge of what actions and what parts are needed in each step and process of the AS.

For example, information for each stage of repair, such as the need to open the bonnet to replace parts A and which wires must be removed, as well as providing information based on digital twins can be visualized.

Through this configuration, the processor provides a modeling image overlap function to the mechanic during repair, minimizes the disassembly area, and minimizes the combination/restoration process accurately, thereby improving the accuracy of the work and significantly reducing the working time. there will be

As an embodiment, the processor may provide the AS status according to the progress of the AS for the simulation target vehicle to the terminal of the owner of the simulation target vehicle.

The processor may provide WebRTC (Video, Voice Description), a vehicle status technician's comment according to the AS stage, as a real-time notification.

Through the above configurations, the digital twin-based vehicle AS simulation device according to an embodiment of the present invention enables accurate and efficient work by AS personnel, as well as providing the AS process to the vehicle owner (customer), transparent and You will be able to proceed with reliable AS.

The method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a server, which is hardware, and stored in a medium.

The aforementioned program is C, C++, JAVA, machine language, etc. It may include a code coded in a computer language of. These codes may include functional codes related to functions defining necessary functions for executing the methods, and include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include memory reference related codes for which location (address address) of the computer's internal or external memory should be referenced for additional information or media required for the computer's processor to execute the functions. there is. In addition, when the processor of the computer needs to communicate with any other remote computer or server in order to execute the functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes for whether to communicate, what kind of information or media to transmit/receive during communication, and the like.

The storage medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and is readable by a device. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers accessible by the computer or various recording media on the user's computer. In addition, the medium may be distributed to computer systems connected through a network, and computer readable codes may be stored in a distributed manner.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

Claims (3)

As a method performed by the vehicle AS simulation device,
Receiving at least one image of a simulation target vehicle;
analyzing the received image and performing a virtual AS simulation for the simulation target vehicle;
deriving simulation data for the simulation target vehicle based on the virtual AS simulation result;
Generating and outputting visualization information based on the derived simulation data,
Vehicle AS simulation method based on digital twin.
According to claim 1,
The vehicle AS simulation device,
Characterized in that the generated visualization information is output based on the digital twin,
Vehicle AS simulation method based on digital twin.
According to claim 1,
Further comprising providing the owner of the simulation target vehicle with the AS status according to the AS progress for the simulation target vehicle.
Vehicle AS simulation method based on digital twin.