KR102529429B1 - Vehicle management assistant method and vehicle management assistant apparatus - Google Patents

Abstract

In one aspect of the present invention, an image acquisition unit for obtaining a vehicle image obtained by photographing the exterior of a vehicle, an additional information acquisition unit for acquiring additional information related to characteristics of the vehicle, and learning to obtain vehicle information based on the vehicle image and a processing unit that obtains vehicle information using the neural network model, and the processing unit obtains a target vehicle image obtained by photographing the outside of the target vehicle through an image acquisition unit, and acquires a target related to the characteristics of the target vehicle through an additional information acquisition unit. A vehicle management assistance device for obtaining additional information and obtaining target vehicle information on a target by using a neural network model based on a target vehicle image.

Description

Vehicle management assistance method and vehicle management assistance device

The present invention relates to a vehicle management assistance method and vehicle management assistance device, and more particularly, to a vehicle management assistance method and vehicle management assistance device for managing exterior state information of a vehicle.

As the economy develops and society advances, the sales volume of vehicles continues to increase. Accordingly, industries related to vehicles, such as parking lots, used car transactions, car insurance, rental cars, and shared car management, are gradually diversifying, and the size of the market is also increasing day by day.

As the size of the relevant market increases, the need to evaluate the condition or value of a vehicle and to continuously manage the vehicle is also increasing due to the needs of various consumers. In particular, if quality control fails due to vehicle damage, etc., this may lead to a decrease in the value of the vehicle as well as a human accident. However, since it is common to mobilize experts for quality control of vehicles, it is a situation where accessibility is insufficient for general consumers.

Accordingly, there is a need to develop a technology that enables general consumers to easily manage vehicle conditions in various industrial environments.

One object of the present invention is to provide a method and apparatus for managing the state of a vehicle.

Another object of the present invention is to provide a method and apparatus for improving the accuracy of vehicle information obtained through a vehicle image.

The problem to be solved by the present invention is not limited to the above-mentioned problems, and problems not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings. .

According to one aspect of the present invention, an image acquisition unit for obtaining a vehicle image, an additional information acquisition unit for acquiring additional information related to characteristics of the vehicle, and a processing unit for obtaining vehicle information based on the vehicle image and the additional information. A vehicle management assisting method for acquiring information related to the exterior of the vehicle based on the vehicle image by using a vehicle management assisting device, comprising: acquiring a target vehicle image for the target vehicle through the image acquisition unit; Acquiring target additional information related to the characteristics of the target vehicle through an additional information acquisition unit and acquiring target vehicle information about the target vehicle by a processing unit based on the target additional information and the target vehicle image, The acquiring of the vehicle information on the target vehicle may include using a neural network model learned to obtain the vehicle information based on the target vehicle image based on the target vehicle image, and using the target vehicle for the target vehicle. information, wherein the neural network model is trained to obtain a first feature set based on the vehicle image and includes a first portion comprising at least one convolutional neural network layer and the first feature set and the additional feature set. A vehicle management assistance method may be provided that includes a second portion that is learned to acquire a second set of features based on information.

According to another aspect of the present invention, in a vehicle management assistance device for acquiring vehicle information related to the exterior of a vehicle based on a vehicle image, an image acquisition unit for acquiring the vehicle image obtained by photographing the exterior of the vehicle, the A processing unit for obtaining vehicle information using a neural network model learned to acquire vehicle information based on vehicle images, and an additional information acquisition unit acquiring additional information related to characteristics of the vehicle, wherein the neural network model comprises the vehicle image A first portion learned to obtain a first feature set based on and including at least one convolutional neural network layer and a second portion learned to obtain a second feature set based on the first feature set and the additional information. The processing unit obtains a target vehicle image acquired by photographing the exterior of the target vehicle through the image acquisition unit, obtains target additional information related to characteristics of the target vehicle through the additional information acquisition unit, and obtains the target vehicle through the additional information acquisition unit. A vehicle management assistance device may be provided that obtains target vehicle information for the target by using the neural network model based on the image.

The solutions to the problems of the present invention are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings. You will be able to.

According to the present invention, a vehicle management assistance method and vehicle management assistance apparatus for effectively managing the state of a vehicle may be provided.

According to the present invention, a vehicle management assistance method and vehicle management assistance apparatus for managing a damaged state of a vehicle can be obtained.

Effects of the present invention are not limited to the above-mentioned effects, and effects not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings.

1 is a diagram for explaining a vehicle management method according to an exemplary embodiment.
2 is a diagram for describing a vehicle image according to an exemplary embodiment.
3 is a diagram for illustrating image data according to an exemplary embodiment.
4 is a diagram for illustrating image data according to an exemplary embodiment.
5 is a diagram for illustrating image data according to another exemplary embodiment.
6 is a diagram for illustrating image data according to an exemplary embodiment.
7 is a diagram for explaining noise elimination processing.
8 is a diagram for illustrating noise removal processing using a plurality of images.
9 is a diagram for illustrating noise removal processing using a plurality of images.
10 is a diagram for explaining a neural network model for obtaining vehicle information according to an exemplary embodiment.
11 is a diagram for explaining a neural network model for obtaining vehicle information according to an exemplary embodiment.
12 is a diagram for explaining a neural network model for obtaining vehicle information according to an exemplary embodiment.
13 is a diagram for illustrating a neural network model according to an embodiment.
14 is a diagram for explaining part area information according to an exemplary embodiment.
15 is a diagram for explaining a vehicle image and part information.
16 is a diagram for explaining a vehicle image and part information.
17 is a diagram for illustrating part learning data according to an embodiment.
18 is a diagram for illustrating part learning data according to another embodiment.
19 is a diagram for explaining damage information according to an exemplary embodiment.
20 is a diagram for explaining damaged area information.
21 is a diagram for illustrating impairment learning data.
22 is a diagram for explaining a device for acquiring part-damage information.
23 is a diagram for explaining acquisition of vehicle information according to an exemplary embodiment.
24 is a diagram for explaining a device for acquiring part-damage information.
25 is a diagram for explaining a device for acquiring part-damage information.
26 is a diagram for explaining new damage information according to an embodiment.
27 is a diagram for explaining damage history management.
28 is a diagram for explaining an output screen according to an exemplary embodiment.
29 is a diagram for explaining display of damaged area information according to an exemplary embodiment.
30 is a diagram for explaining a vehicle management assistance method according to an exemplary embodiment.
31 is a diagram for explaining a vehicle damage management method according to an embodiment.
32 is a diagram for explaining a vehicle damage management method according to an embodiment.
33 is a diagram for explaining a vehicle damage management method according to an embodiment.
34 is a diagram for explaining the configuration of an on-site vehicle management system.
35 is a diagram for explaining a series of steps performed by the on-site vehicle management process.
36 is a diagram for explaining a step of acquiring vehicle data necessary for vehicle management in an on-site vehicle management process.
37 is a diagram for explaining that image data is captured by a camera module at a specific point in time when image data is sensed by a sensing unit according to an exemplary embodiment.
38 is a diagram for explaining that image data is captured by a camera module at a specific point in time when image data is sensed by a sensor according to another embodiment.
39 is a diagram for explaining that image data is captured by a camera module at a specific point in time when image data is sensed by a sensor according to another embodiment.
40 is a diagram for explaining that image data is captured by a camera module at a time point within a certain range from a time point at which a circuit breaker operates according to another embodiment.
41 is a diagram for explaining a method of extracting image data from image data according to an exemplary embodiment.
42 is a diagram for illustrating noise removal processing using a plurality of images.
43 is a diagram for illustrating noise removal processing using a plurality of images.
44 is a view for explaining a screen for outputting damage information among output screens of a management device.
45 is a view for explaining a screen for selecting whether a user or a manager consents to damage information among output screens of a management device.
46 is a view for explaining a screen including detailed information on damage information among output screens of a management device.
47 is a diagram for explaining an output screen including information about a damage history among output screens of a management device.
48 is a view for explaining an output screen including information about a damage history among output screens of a management device.
49 is a diagram for explaining the configuration of a mobile-based vehicle management system.
50 is a diagram for explaining a series of steps performed by a mobile-based vehicle management process.
51 is a diagram for explaining a recapturing request when vehicle damage information included in image data is not sufficient, according to an exemplary embodiment.
52 is a diagram for explaining that image data is captured by a camera module at a specific point in time according to an exemplary embodiment.
53 is a view for explaining a photographing guide provided based on target vehicle information when photographing a vehicle with a camera module.
54 is a view for explaining a photographing map provided when a vehicle is photographed with a camera module.
55 is a diagram for explaining a screen for outputting vehicle information and/or auxiliary information in a terminal according to an exemplary embodiment.
56 is a diagram schematically illustrating a process in which a plurality of shared vehicle users use a shared vehicle and end the use according to a time sequence.
57 is a view for explaining a vehicle return process in which a shared vehicle user photographs the exterior of the vehicle after using the shared vehicle and checks additional damage information.
58 is a view for explaining a vehicle use process in which a shared vehicle user indicates whether or not to agree to existing damage information of a shared vehicle before using the shared vehicle and, if not, takes a photograph of the exterior of the vehicle.
59 is a diagram for explaining a process of re-photographing the exterior of a shared vehicle when a shared vehicle user does not agree with existing damage information of the shared vehicle.

The foregoing objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. However, the present invention can apply various changes and can have various embodiments. Hereinafter, specific embodiments will be illustrated in the drawings and described in detail.

In the drawings, the thickness of layers and regions is exaggerated for clarity, and an element or layer is referred to as "on" or "on" another element or layer. It includes all cases in which another layer or other component is intervened in the middle as well as immediately above another component or layer. Like reference numerals designate essentially like elements throughout the specification. In addition, components having the same function within the scope of the same idea appearing in the drawings of each embodiment are described using the same reference numerals.

If it is determined that a detailed description of a known function or configuration related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of this specification are only identification symbols for distinguishing one component from another component.

In addition, the suffixes "module" and "unit" for the components used in the following description are given or used interchangeably in consideration of only the ease of writing the specification, and do not have meanings or roles that are distinguished from each other by themselves.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

According to an example, a vehicle management assistance device for obtaining vehicle information related to the exterior of a vehicle based on a vehicle image includes: an image acquisition unit configured to acquire the vehicle image obtained by photographing the exterior of the vehicle; an additional information obtaining unit acquiring additional information related to the characteristics of the vehicle; and a processing unit acquiring the vehicle information by using a neural network model learned to obtain the vehicle information based on the vehicle image. Wherein, the neural network model is learned to obtain a first feature set based on the vehicle image and a first part including at least one convolutional neural network layer and a first feature set based on the first feature set and the additional information 2 a second part learning to acquire a set of features, wherein the processing unit acquires a target vehicle image obtained by photographing the outside of the target vehicle through the image acquisition unit, and acquires characteristics of the target vehicle through the additional information acquisition unit. It is possible to provide a vehicle management assistance device that obtains target additional information related to and acquires target vehicle information on the target using the neural network model based on the target vehicle image.

The neural network model may be trained to acquire damage information related to damage to the exterior of the vehicle, and the second feature set may include at least one feature value related to whether or not the exterior of the vehicle is damaged.

The neural network model may be trained to acquire part information related to a part constituting the exterior of the vehicle, and the second feature set may include at least one feature value related to the part constituting the exterior of the vehicle.

The additional information related to the characteristics of the vehicle may be determined based on at least one of a manufacturer of the vehicle, color information, a model year, and a vehicle type.

The additional information related to the characteristics of the vehicle may be obtained based on the vehicle interior image obtained by photographing the interior of the vehicle.

The second part obtains the second feature set related to at least one part constituting the exterior of the vehicle based on the first feature set and the additional information, the second feature set being the first part. A first region feature set representing the distribution area and a second region feature set representing the distribution region of the second component may be included.

The second part obtains the second feature set related to at least one damaged area located outside the target vehicle based on the first feature set and the additional information, the second feature set comprising the damage It may include a set of damaged area features indicating the location of the area.

According to an example, vehicle management including an image acquisition unit acquiring a vehicle image, an additional information acquisition unit acquiring additional information related to characteristics of the vehicle, and a processing unit acquiring vehicle information based on the vehicle image and the additional information. A vehicle management assisting method for obtaining information related to the exterior of the vehicle based on the vehicle image by using an auxiliary device, comprising: acquiring a target vehicle image of the target vehicle through the image acquiring unit; obtaining target additional information related to characteristics of the target vehicle through the additional information obtaining unit; and acquiring, by the processing unit, target vehicle information on the target vehicle based on the additional target information and the target vehicle image. Including, the obtaining of the vehicle information on the target vehicle comprises, based on the target vehicle image, using a neural network model learned to obtain the vehicle information based on the vehicle image, further comprising obtaining the target vehicle information for, wherein the neural network model is learned to obtain a first feature set based on the vehicle image and a first part comprising at least one convolutional neural network layer and the first feature and a second part learning to acquire a second feature set based on the set and the additional information.

The neural network model may be trained to acquire damage information related to damage to the exterior of the vehicle, and the second feature set may include at least one feature value related to whether or not the exterior of the vehicle is damaged.

The neural network model may be trained to acquire part information related to a part constituting the exterior of the vehicle, and the second feature set may include at least one feature value related to the part constituting the exterior of the vehicle.

The additional information related to the characteristics of the vehicle may be determined based on at least one of a manufacturer, color information, year, and model of the vehicle.

Additional information related to the characteristics of the vehicle may be obtained based on the vehicle interior image obtained by photographing the interior of the vehicle.

The second part obtains the second feature set related to at least one part constituting the exterior of the target vehicle based on the first feature set and the additional information, wherein the second feature set is the first feature set. A first region feature set indicating a region where parts are distributed and a second region feature set indicating a region where the second part is distributed may be included.

The second part obtains the second feature set related to at least one damaged area located outside the target vehicle based on the first feature set and the additional information, the second feature set comprising the damage It may include a set of damaged area features indicating the location of the area.

A computer readable recording medium storing a program for performing the vehicle management assistance method may be provided.

According to one example, an output unit for outputting a damaged image; And a control unit for controlling the output unit to output the damaged image based on a first image captured by a camera at a first point of view and a second image captured by the camera at a second point in time, wherein the first The image is taken of a first area related to the exterior of the vehicle, and at least one fixed area and a variable area are included inside the first area, and the second image is a second area related to the exterior of the vehicle. is captured, and at least a partial area of the fixed area and at least a partial area of the variable area are included inside the second area, and the variable area is removed from the first image and the second image in the damaged image. However, the fixed area is an area for damage to the vehicle, the variable area is an area other than the area for damage to the vehicle, and the first and second views are different views. can

The first image and the second image include a first feature point among a plurality of feature points related to a vehicle, the first feature point of the first image has a first coordinate, and the first feature point of the second image has a first feature point. 2 coordinates, the coordinate representing the fixed area included in the first image is a third coordinate, the coordinate representing the fixed area included in the second image is a fourth coordinate, and the first coordinate and A first distance between third coordinates and a second distance between the second coordinate and the fourth coordinate may correspond to each other.

The coordinate representing the variable region included in the first image is a fifth coordinate, the coordinate representing the variable region included in the second image is a sixth coordinate, and the distance between the first coordinate and the fifth coordinate is The third distance may provide a vehicle management device different from the fourth distance between the second coordinate and the sixth coordinate.

A difference between the first distance and the second distance may be smaller than a difference between the third distance and the fourth distance.

The control unit classifies a fixed area and a variable area in the abnormal area and outputs the damaged image from which the variable area is removed, wherein the first image and the second image include a first feature point among a plurality of feature points related to the vehicle, , the first feature point of the first image has a first coordinate, the first feature point of the second image has a second coordinate, and the coordinate representing the abnormal region of the first image is a seventh coordinate, Coordinates representing the abnormal region of the second image are eighth coordinates, and the controller determines a fifth distance between the first coordinate and the seventh coordinate and a sixth distance between the second coordinate and the eighth coordinate. Based on the above, it is possible to provide a vehicle management device that classifies the fixed area and the variable area.

Wherein the control unit determines a variable area when the difference between the fifth distance and the sixth distance is equal to or greater than a predetermined value, and determines a fixed area when the difference between the fifth distance and the sixth distance is less than a predetermined value, the vehicle management device can provide

The variable area may provide a vehicle management apparatus that is a background area outside the vehicle.

The variable area may provide a vehicle management device that is an area included in an image where an external object is reflected by external light.

The damaged image is an image in which the variable region is removed from the first image, and the first image is an image in which the vehicle is located closer to the central region of the image than the second image.

A communication unit for receiving a signal from the sensor unit is further included, the sensor unit includes a first sensor and a second sensor, the first sensor is located in a region close to the vehicle entry direction, and the second sensor is the vehicle entry direction. It is possible to provide a vehicle management device located in an area far from the direction.

The first time point and the second time point are points within an effective period between a start point and an end point, the start point is a point in time when the communication unit receives the first signal from the first sensor, and the end point is a point in time when the communication unit receives the first signal. A time point at which the second signal is received from the second sensor. The first image and the second image may provide a vehicle management device included in a plurality of images obtained within the valid period.

The start time is a time when the vehicle is detected by the first sensor and then the vehicle detection is ended, and the end time is a time when the vehicle is first detected by the second sensor.

The starting time point is a time point when the vehicle is initially detected by the first sensor, and the end time point is a time point when the vehicle is sensed by the second sensor and then the sensing is finished.

A communication unit for receiving open information from the circuit breaker, wherein the first time point and the second time point are time points within an effective period between a start time point and an end time point, the time point at which the circuit breaker is opened is an end time point, and the time point at which the circuit breaker is opened A time point prior to the specified time point by a predefined time point is a starting time point, and the first image and the second image are included in a plurality of images obtained within the valid period.

The camera includes a first camera and a second camera, the first camera is located in an area corresponding to the first sensor, and photographs one side of the vehicle, and the second camera corresponds to the second sensor. It is possible to provide a vehicle management device that is located in an area where the vehicle is located and takes pictures of the other side of the vehicle.

The first image and the second image include a first feature point among a plurality of feature points related to the vehicle, the camera is located at a position having a first distance from the first feature point at the first point of view, and at the second point of view The camera may be located at a position having a second distance from the first feature point, and the first distance and the second distance may be different from each other.

The vehicle management device may be a mobile device, and the camera may be a camera built into the mobile device.

The camera may capture a video between a start time and an end time, and the first image and the second image may be images extracted from the video.

According to an example, an apparatus for managing damage to a vehicle that acquires information related to external damage of a vehicle based on a vehicle image includes: an image acquisition unit that acquires at least one target vehicle image obtained by photographing an exterior of a target vehicle; and a processing unit obtaining damage information related to external damage of the target vehicle based on the at least one target vehicle image. The at least one target vehicle image includes a first target vehicle image obtained by photographing one side of the target vehicle, wherein the first target vehicle image comprises a first region corresponding to the target vehicle and the A second area in which identification information of a target vehicle is located—the first area is greater than the second area—, wherein the processing unit determines vehicle type information of the target vehicle based on the second area and the vehicle type information and a vehicle damage management apparatus for obtaining the damage information obtained based on the first target vehicle image and related to the first region.

an output unit outputting the damage information; Further comprising, but the output unit may provide a vehicle damage management device that outputs the damage information of the target vehicle.

The damage information includes damage area information corresponding to the damage of the target vehicle, and the output unit outputs the target vehicle image and the damage area information superimposed on the target vehicle image. .

An input unit that obtains a user input; and the processing unit obtains a user response to the damage information through the input unit in response to an output of the damage information.

The processing unit may provide a vehicle damage management apparatus that obtains identification information of the target vehicle determined based on the second region.

The processing unit obtains identification information of the target vehicle determined based on the second area, obtains a damage history of the target vehicle corresponding to the identification information of the target vehicle, and the damage history is determined based on the first target vehicle. It is possible to provide a vehicle damage management apparatus that is obtained based on a second target vehicle image taken prior to a vehicle image and includes existing damage information related to the exterior damage of the target vehicle.

The processing unit may provide a vehicle damage management device that obtains new damage information corresponding to a time point when the first target vehicle image of the target vehicle was captured, based on the existing damage information and the damage information.

The damage information may include first damage information related to the second area included in the first target vehicle image and second damage information previously stored in association with identification information of the target vehicle obtained based on the first area. It is possible to provide a vehicle damage management device that is created based on the above.

The at least one target vehicle image further includes a second target vehicle image obtained by photographing the other side of the target vehicle, wherein the second target vehicle image includes a third region corresponding to the target vehicle. and the processing unit determines vehicle type information of the target vehicle based on the second area of the first target vehicle image and the damage obtained based on the vehicle type information and the second target vehicle image and related to the third area. It is possible to provide a vehicle damage management device that acquires information.

The processing unit may, based on the first target region image and the vehicle type information, a first part region included in the first region and corresponding to a first part of the target vehicle and the target vehicle included in the first region It is possible to provide an apparatus for managing damage to a vehicle that obtains part information including a second part area corresponding to the second part of the vehicle.

The processing unit may provide an apparatus for managing damage to a vehicle that obtains part area information corresponding to the damage based on the damage information and the part information.

The processing unit obtains the second area corresponding to the license plate of the target vehicle from the first target vehicle image, and provides a vehicle damage management method for obtaining the vehicle model information based on the second area corresponding to the license plate. can do.

The processing unit may provide a vehicle damage management method for obtaining the damage information based on the target vehicle image by using a neural network model learned to acquire the damage information based on the target vehicle image.

The damage information may provide a vehicle damage management apparatus including region information indicating a region corresponding to damage outside the vehicle among the first region.

According to an example, in a vehicle damage management method using an apparatus including an image acquisition unit acquiring a vehicle image and a processing unit obtaining damage information, at least one obtained by photographing the exterior of a target vehicle through the image acquisition unit. Obtaining a target vehicle image of the at least one target vehicle image includes a first target vehicle image obtained by photographing one side of the target vehicle, wherein the first target vehicle image corresponds to the target vehicle including a first area and a second area where identification information of the target vehicle is located; acquiring vehicle type information of the target vehicle determined based on the second area through the processing unit; and acquiring damage information related to exterior damage of the target vehicle based on vehicle model information and the first target vehicle image through the processing unit. It is possible to provide a vehicle damage management method comprising a.

outputting the damage information; Further comprising, wherein the damage information includes damage area information corresponding to the damage of the target vehicle, and outputting the damage information outputs the damage area information overlapping the target vehicle image and the target vehicle image. It is possible to provide a vehicle damage management method further comprising.

According to an example, obtaining identification information of the target vehicle determined based on the second region through the processing unit; Further comprising, it is possible to provide a vehicle damage management method.

Acquiring, through the processing unit, a damage history of the target vehicle corresponding to the identification information of the target vehicle - the damage history is obtained based on a second target vehicle image captured prior to the first target vehicle image, and It is possible to provide a vehicle damage management method that further includes; including existing damage information related to the exterior damage of the target vehicle.

obtaining, through the processing unit, new damage information corresponding to a point in time when the first target vehicle image of the target vehicle was captured, based on the existing damage information and the damage information; It is possible to provide a vehicle damage management method further comprising.

The damage information may provide a vehicle damage management method including damage area information included in the first area and corresponding to the damage of the target vehicle.

The at least one target vehicle image further includes a second target vehicle image obtained by photographing the other side of the target vehicle, wherein the second target vehicle image includes a third region corresponding to the target vehicle. The obtaining of the damage information may include obtaining the damage information based on vehicle model information of the target vehicle determined based on the second area of the first target vehicle image and the vehicle model information and the second target vehicle image, and A vehicle damage management method may further include acquiring the damage information related to the third region.

Through the processing unit, based on the first target area image and the vehicle model information, a first part area included in the first area and corresponding to the first part of the target vehicle and the target included in the first area obtaining parts information including a second part area corresponding to a second part of the vehicle; It is possible to provide a vehicle damage management method further comprising.

The vehicle damage management method may further include obtaining part area information corresponding to the damage based on the damage information and the parts information through the processing unit.

obtaining the second area corresponding to the license plate of the target vehicle from the first target vehicle image through the processing unit; and obtaining the vehicle type information may further include obtaining the vehicle type information based on the second area corresponding to the license plate.

The acquiring of the damage information further comprises obtaining the damage information based on the target vehicle image by using a neural network model learned to obtain the damage information based on the target vehicle image. can provide.

The vehicle damage management apparatus further includes an input unit for obtaining a user input, and obtaining a user response to the damage information through the input unit in response to the processing unit outputting the damage information; It is possible to provide a vehicle damage management method further comprising.

For example, a computer readable recording medium storing a program for performing the vehicle damage management method may be provided.

For example, a camera for taking an image of a vehicle; an input unit that receives information from a user; Communication unit for transmitting and receiving radio signals; a display unit displaying a message related to existing damage information; and a controller for controlling the camera, the communication unit, and the display unit, wherein the user includes a previous user and a current user, and the controller provides the existing damage information to the current user through the display unit, and the existing damage information If the current user agrees to, the vehicle is changed so that the vehicle can be used, and the existing damage information is damage information calculated based on a first image taken by the previous user, and the display unit is the existing damage information. It is possible to provide a mobile device for displaying a first notification window through which the current user can select consent or not.

The first image may be an image taken by the previous user when returning the vehicle, and the existing damage information may indicate damage information generated until the previous user ends using the vehicle.

The damage information may provide a mobile device that is information updated by the first image transmitted by the mobile device of the previous user.

The display unit displays a second notification window requesting the current user to capture a second image when the current user does not agree with the existing damage information, and the controller displays the second notification window when capturing the second image is completed. A mobile device that transmits an image to a server through the communication unit and changes the vehicle into a usable state may be provided.

When the current user does not agree with all of the existing damage information, the display unit displays a second notification window requesting the current user to capture a second image of the entire exterior of the vehicle, and the control unit displays the second image When photographing is completed, a mobile device may be provided that transmits the second image to the server through the communication unit and changes the vehicle to a usable state.

When the current user does not agree with some of the existing damage information, the display unit displays a second notification window requesting the current user to take a second image of a part of the exterior of the vehicle to which the current user does not agree, and , The control unit may provide a mobile device that transmits the second image to the server through the communication unit when capturing the second image is completed and changes the vehicle to a usable state.

The display unit displays a photographing guide, and the photographing guide guides the entire exterior of the vehicle or parts of the vehicle to be photographed when the current user captures the second image. there is.

The photographing guide may guide photographing based on vehicle information, and the vehicle information may be obtained through information input by the current user or a neural network model trained to obtain vehicle information. can

The control unit may provide a mobile device that automatically executes a photographing button and stores the image when the vehicle is located within a predetermined ratio or more in the photographing guide when the current user captures the second image.

wherein the display unit displays a photographing map, and the photographing map guides photographing so that at least a part of main parts of the exterior of the vehicle are included in the second image when the current user photographs the second image. device can be provided.

The control unit receives a comparison result obtained by the server comparing the second image and the existing damage information and determining whether additional damage has occurred to the vehicle through the communication unit, and the control unit transmits the comparison result through the display unit to the current damage information. Provided to the user, the display unit may provide a mobile device for displaying a third notification window through which the current user can select whether to consent to the comparison result.

The control unit may provide a mobile device that changes the vehicle into a usable state when the current user agrees with the comparison result.

The display unit displays a message for making an objection if the current user does not agree with the comparison result, and the control unit changes the vehicle to become usable when the current user completes the objection. device can be provided.

For example, a camera for taking an image of a vehicle; an input unit that receives information from a user; Communication unit for transmitting and receiving radio signals; a display unit displaying a message related to vehicle damage information; and a control unit controlling the camera, communication unit, and display unit, wherein the user includes a previous user and a current user, and the display unit requests the current user to take a third image when returning the vehicle of the current user. A fourth notification window may be displayed, and the control unit may transmit the third image to a server through the communication unit when capturing of the third image is completed, and process the vehicle return process to be completed.

The display unit displays a photographing guide, and the photographing guide guides the entire vehicle or parts of the vehicle to be photographed when the current user captures the third image. there is.

wherein the display unit displays a photographing map, and the photographing map guides photographing so that at least a part of main parts of the exterior of the vehicle may be included in the second image when the current user photographs the third image. device can be provided.

The control unit receives additional damage information determined by the server comparing the third image with existing damage information to determine whether additional damage has occurred to the vehicle through the communication unit, and the control unit transmits the additional damage information through the display unit. It is provided to the current user, and the display unit may provide a mobile device for displaying a fifth notification window through which the current user may select whether to consent to the additional damage information.

The additional damage information may provide a mobile device that means damage information generated from the start of the current user's vehicle use to the end of the current user's vehicle use.

The control unit may provide a mobile device that processes the vehicle return process to be completed when the current user agrees to the additional damage information.

If the current user does not agree with the additional damage information, the display unit displays a message for making an objection, and the controller controls the vehicle return process to be completed when the current user completes the objection. A mobile device for processing may be provided.

Hereinafter, a vehicle management device according to an embodiment will be described with reference to the drawings.

1. Overview

In this specification, an apparatus, method, and system for managing the state of a vehicle will be described. Hereinafter, a device for managing a state of a vehicle and the like will be described with reference to some embodiments.

2. Vehicle Management System

According to an embodiment, a vehicle management system for managing a state of a vehicle using a photographed image of the vehicle may be provided.

A vehicle management system may include at least one device. The vehicle management system may include a device having at least one camera module. The vehicle management system may include a device having at least one control unit. The vehicle management system may include at least one device including an input module that obtains a user input. The vehicle management system may include at least one device having a storage module. The vehicle management system may include at least one device having a communication module.

The vehicle management system may include user devices such as tablets, smart phones, PCs, and laptops. A user device may include a display, user interface, communication module, camera module, memory and/or controller.

The vehicle management system may include a server device. A server device may store data or programs. A server device may communicate with a user device.

The vehicle management system may include a vehicle photographing device. The vehicle photographing device may communicate with a server device and/or a user device.

2.1 Vehicle Management Process

According to one embodiment, a vehicle management process for managing a state of a vehicle using a photographed image of the vehicle may be provided. The vehicle management process may be performed by the aforementioned vehicle management system.

Referring to FIG. 1 , the vehicle management process acquires vehicle data related to the vehicle (S110), obtains vehicle information based on the obtained vehicle data (S130), manages the vehicle information (S150), and It may include outputting (S170). Hereinafter, each step will be described in more detail.

2.2 Data Acquisition

The vehicle management process may include acquiring vehicle data necessary for vehicle management.

Vehicle data may be prepared in various forms. For example, vehicle data may be image data or non-image data. Vehicle data may be pre-stored in the device or obtained from the outside. For example, vehicle data may be acquired through a camera module or through a user input module.

The acquisition of vehicle data and vehicle data will be described in more detail below with specific examples.

2.2.1 Image data acquisition

According to an embodiment, the vehicle data may be image data including a vehicle image obtained by photographing the vehicle. The vehicle image may be an image including some parts of the vehicle. The vehicle image may be an image obtained by photographing a part of the vehicle to include identification information for identifying the vehicle.

The vehicle data may be vehicle image data obtained by photographing the vehicle. The vehicle data may include image data and image data extracted from the image data. Hereinafter, an embodiment of vehicle data based on image data will be described. However, vehicle data may be acquired in the form of a vehicle image corresponding to a vehicle image of a form described below.

2 is a diagram for describing a vehicle image according to an exemplary embodiment.

Referring to (a) of FIG. 2 , a vehicle image according to an embodiment may be an image obtained by photographing a vehicle in one direction, for example, from the side. The vehicle image may be an image obtained by photographing the entire vehicle in a frame from one direction.

Referring to (b) of FIG. 2 , a vehicle image according to an embodiment may be an image obtained by photographing the vehicle from the front. The vehicle image may include identification information for identifying the vehicle. The identification information may be a Vehicle Identification Number (VIN) or vehicle number. Referring to (b) of FIG. 2 , a vehicle image according to an exemplary embodiment may include a VIN area displaying a vehicle VIN or a license plate area displaying a vehicle number.

Referring to (c) of FIG. 2 , a vehicle image according to an embodiment may be an image captured to include some parts of the vehicle, for example.

According to one embodiment, vehicle data may include a plurality of vehicle images.

A plurality of vehicle images may include one or more images taken at different viewpoints. For example, a plurality of vehicle images may include images of the same vehicle captured at different times. The plurality of vehicle images may include images captured in the same direction (eg, from the front) based on the same vehicle.

3 is a diagram for illustrating image data according to an exemplary embodiment. Referring to FIG. 3 , image data includes a first image (a) of a vehicle taken at a first point of view, a second image (b) of a vehicle taken at a second point of view after the first point of view, and a vehicle at a third point of view. It may include a third image (c) taken. The first image, the second image, and the third image may be images extracted from one image data. The first image, the second image, and the third image may be images captured from the same vehicle in the same direction, for example, from the left side.

The plurality of vehicle images may include one or more images obtained by photographing the vehicle in various directions.

4 is a diagram for illustrating image data according to an exemplary embodiment. Referring to FIG. 4 , image data includes a left image (a) obtained by photographing the left side of the vehicle, a right image (b) obtained by photographing the right side of the vehicle, and a front image (c) obtained by photographing the front of the vehicle. and a rear image (d) obtained by photographing the rear of the vehicle.

5 is a diagram for illustrating image data according to another exemplary embodiment. Referring to FIG. 5 , the image data is a left front image obtained by photographing the left front of the vehicle (a), a left rear image obtained by photographing the left rear of the vehicle (b), and an image obtained by photographing the right front of the vehicle. It may include a right front image (c) and a right rear image (d) obtained by photographing the right rear of the vehicle.

The plurality of vehicle images may include one or more images taken to include different parts of the vehicle.

6 is a diagram for illustrating image data according to an exemplary embodiment. Referring to FIG. 6 , image data includes a first image (a) obtained by photographing to include the left front fender of the vehicle, a second image (b) obtained by photographing to include the left front door of the vehicle, and the left side of the vehicle. A third image (c) obtained by photographing the rear door and a fourth image (d) obtained by photographing the left rear fender of the vehicle may be included.

According to an embodiment, image data obtained by photographing an object other than a vehicle may be obtained. For example, an object image obtained by photographing objects around the vehicle, an image obtained by photographing the surrounding environment of the vehicle, eg, an image obtained by photographing the sky, and an image obtained by photographing a driver's ID card or driver's license may be acquired.

Image data or video data may be acquired through a camera module. Image data or video data may be obtained from an external device, for example, an external device having a camera module.

2.2.2 Acquisition of non-image data

According to one embodiment, vehicle data in the form of non-images other than images may be obtained. Vehicle data may be non-image data. Vehicle data may be non-image data including vehicle-related information.

The vehicle data may include vehicle information indicating the manufacturer, model name, year, manufacturing time, manufacturing place, manufacturer, location, and registration reference place of the vehicle.

The vehicle data may include vehicle model information indicating the type of vehicle. The vehicle type information may indicate a vehicle type such as a sedan, a coupe, a wagon, a convertible, a hatchback, an SUV, a van, a truck, or a bus.

The vehicle data may include vehicle identification information for identifying the vehicle. The vehicle identification information may include a vehicle number assigned to the vehicle, a vehicle identification number (VIN), and the like.

According to one embodiment, additional information in the form of a non-image may be further obtained. For example, additional information including driver information indicating personal information of the driver or vehicle owner, insurance information indicating whether or not the vehicle is insured, insurance type, and the like, and accident history of the vehicle may be further obtained.

Non-image data may be pre-stored or acquired. Non-image data may be obtained through an input module that obtains user input. Non-image data may be obtained through a user interface. Non-image data may be obtained from an external device.

2.2.3 Data processing

Processing may be performed on the acquired data.

Pre-processing (or post-processing) may be performed on the acquired image data. For example, with respect to the acquired image (eg, vehicle image), brightness change, saturation change, color change, color temperature change, luminance change, size change, image cropping, pixel number adjustment, size change, horizontal and vertical ratio adjustment, pixel Pixel linearization, filtering, dead pixel correction, image sharpening, geometric distortion correction, chroma suppression, image scaling, dithering ( Processing such as dithering, interpolation, compression, and decompressing may be performed.

For example, noise removal processing may be performed on the acquired image data. For the vehicle image, denoising processing may be performed. Noise elimination processing may be performed to remove noise such as reflected light and stains located in the background area or vehicle area included in the image, that is, image information unnecessary for vehicle management. Noise removal processing may be performed on the image by performing processing such as a Gaussian filter. Alternatively, with respect to the image, denoising processing may be performed through a neural network model trained for denoising.

7 is a diagram for explaining noise elimination processing.

According to one embodiment, noise removal processing may be performed to remove non-vehicle objects from the vehicle image. Referring to (a) and (b) of FIG. 7 , in the noise removal process according to an exemplary embodiment, an original image (a) including a vehicle area corresponding to a vehicle and a background area not corresponding to the vehicle is obtained, and , obtaining a preprocessed image (b) from which the background region has been removed.

Denoising processing may be performed using a neural network model trained to obtain a vehicle area from an original image. For example, the neural network model may include a convolutional neural network (CNN) layer and be trained to extract a vehicle area from an original image based on training data including a plurality of images in which the vehicle area is masked from the original image. there is. A noise removal process may be performed to extract a vehicle area from the original target image and remove the remaining areas using the learned neural network model.

According to another embodiment, noise removal processing may be performed to remove elements that interfere with acquisition of vehicle information from a vehicle image. The noise removal process for removing elements obstructing vehicle information acquisition may include removing noise included in a vehicle area corresponding to the vehicle. Referring to (c) and (d) of FIG. 7 , noise removal processing according to an embodiment acquires a vehicle image for acquiring vehicle damage information, and corresponds to reflected light from a vehicle area unrelated to vehicle damage. It may include removing the specular light region.

The denoising process may be performed using a neural network model trained to obtain an image from which reflected light has been removed from the original image. For example, the neural network model includes a convolutional neural network layer, and based on training data including a plurality of images in which the specular region is masked from the original image (or the original image including the specular region and the image from which the specular region is manually removed) , it can be learned to obtain an image from which the specular region is removed from the original image. A noise removal process may be performed to obtain an image from which specular regions are removed from the target original image by using the learned neural network model.

As another example, noise removal processing may be performed on image data including a plurality of images. When the image data includes a plurality of images, noise removal processing may be performed on each image or noise removal processing with improved accuracy may be performed by using one or more images together.

Based on the plurality of original images, noise removal processing may be performed to obtain an image from which the background area is removed. The noise removal process acquires a plurality of images, distinguishes between an object whose position is changed (eg, an object corresponding to a vehicle) and an object whose position is not changed (eg, an object corresponding to a background), and an object whose position is changed. Alternatively, it may include removing an object whose location does not change. The position change may mean a relative position change within an image frame. For example, the noise removal process may include acquiring a plurality of images, removing objects whose positions do not change, and obtaining an image including only objects whose positions change. Also, for example, the noise removal process may include acquiring a plurality of images, removing an object whose position is fluctuated, and acquiring an image including only an object whose position is not fluctuated. Here, an object whose location changes may be defined as a variable area, and an object whose location does not change may be defined as a fixed area.

8 is a diagram for illustrating noise removal processing using a plurality of images. Referring to FIG. 8 , a plurality of original images (a, b, and c) having a vehicle area and a background area are obtained, and an image (d) from which the background area is removed based on the plurality of original images (a, b, and c) is obtained. ) may be performed. Referring to FIG. 8 , based on a plurality of original images (a, b, and c), an area corresponding to a background (ie, a background area), which is an object whose position is fixed in the image, is removed, and An image d including an area corresponding to a vehicle, which is an object whose location is changed (ie, a vehicle area), may be acquired.

The background-removed image (d) may be generated based on any one of the plurality of original images (a, b, and c). For example, the denoising process obtains a background area and a vehicle area based on a plurality of original images (a, b, and c), removes the background area from any one of the plurality of original images (a, b, and c), It may include obtaining a denoised image (d).

Here, noise removal processing may be performed based on at least two images among the plurality of original images a, b, and c. The image on which the noise removal process is performed may be defined as a first image and a second image.

For example, (a) of FIG. 8 may be defined as a first image, and (b) of FIG. 8 may be defined as a second image. The first image and the second image may be images taken at different times. The first image may be an image captured at a first viewpoint, and the second image may be an image captured at a second viewpoint. The first viewpoint and the second viewpoint may be different from each other.

Denoising processing may be performed using a neural network model. The neural network model may include a convolutional neural network layer and may be trained to obtain an image from which the background area is removed using training data including a plurality of vehicle images in which the vehicle area and/or the background area are masked. The denoising process may include obtaining a vehicle image from which a background region is removed from target original images by using a trained neural network model.

Based on the plurality of original images, noise removal processing for removing reflected light or the like may be performed. The noise removal process acquires a plurality of images, and distinguishes between an object whose position changes (eg, a reflective area corresponding to reflected light) and an object whose position does not change (eg, a vehicle area or a damaged area corresponding to damage to a vehicle). Thus, it may include removing an object whose location is changed or an object whose location is not changed. For example, based on a vehicle image including a damaged area corresponding to a damaged part of the vehicle and a reflective area corresponding to light reflected by the surface of the vehicle, noise removal processing may be performed to obtain a vehicle image from which the reflective area is removed. . Here, an object whose location changes may be defined as a variable area, and an object whose location does not change may be defined as a fixed area. That is, the damaged area may be defined as a fixed area, and the reflective area may be defined as a variable area.

9 is a diagram for illustrating noise removal processing using a plurality of images. Referring to FIG. 9 , noise removal processing may be performed to obtain a plurality of original images (a and b) including a damaged area and a reflective area, and to obtain an image (c) from which the reflective area is removed. Referring to FIG. 9 , based on a plurality of original images (a and b), noise removal processing is performed to obtain an image (c) in which a reflective area whose position or state is changed in an image or in a vehicle area is removed. It can be.

The image (c) from which the reflection area is removed may be generated based on any one of the plurality of original images (a and b). For example, the denoising process is to obtain a reflective area based on a plurality of original images (a and b), remove the reflective area from any one of the plurality of original images (a and b), and obtain a denoised image. may include

Here, noise removal processing may be performed based on at least two images among the plurality of original images a, b, and c. The image on which the noise removal process is performed may be defined as a first image and a second image.

For example, (a) of FIG. 9 may be defined as a first image, and (b) of FIG. 9 may be defined as a second image. The first image and the second image may be images taken at different times. The first image may be an image captured at a first viewpoint, and the second image may be an image captured at a second viewpoint. The first viewpoint and the second viewpoint may be different from each other.

The noise removal process may be performed based on a difference between positions of the fixed and variable regions in the first image and positions of the fixed and variable regions in the second image.

In the noise removal process, a fixed region and a variable region are classified according to a distance difference between the abnormal regions included in the first image and the second image, and the variable region is removed from the classified fixed and variable regions to create a damaged image. can be processed by The damaged image may be an image obtained by removing a variable region from an image in which the vehicle is located close to the center region of the first image and the second image.

For example, the first image includes a plurality of abnormal areas including a first abnormal area and a second abnormal area, and the second image includes a plurality of abnormal areas including a third abnormal area and a fourth abnormal area. Including, it is defined that the first abnormal area corresponds to the third abnormal area, and the second abnormal area corresponds to the fourth abnormal area.

The denoising process may classify a fixed region and a variable region based on the position of each abnormal region in the first image and the position of each abnormal region in the second image.

The position of the abnormal region may be a relative position defined based on a partial region of the vehicle. That is, when any one of the plurality of feature points of the vehicle included in the first image and the second image is defined as the first feature point, the first feature point located in the abnormal area may be defined as a reference.

The first feature point may have different coordinates in each image.

For example, the first feature point in the first image may have a first coordinate, and the first feature point in the second image may have a second coordinate.

Since the ideal area is not a single point but an area having an area, coordinates representing each area must be defined for comparison with the first feature point. In the noise removal process, coordinates representing each region may be defined. The noise removal process may be defined as a coordinate representing one of a plurality of coordinates included in each region. For example, in the noise removal process, coordinates of the center of each region may be defined as coordinates representing the corresponding region.

Coordinates representing the first ideal region of the first image may be defined as third coordinates, and coordinates representing the second ideal region may be defined as fourth coordinates.

Also, coordinates representing the third ideal region in the second image may be defined as fifth coordinates, and coordinates representing the fourth ideal region may be defined as sixth coordinates.

A first distance between the first coordinate and the third coordinate and a second distance between the second coordinate and the fifth coordinate may correspond to each other. A third distance between the first coordinate and the fourth coordinate may be different from a fourth distance between the second coordinate and the sixth coordinate.

A difference between the first distance and the second distance may be smaller than a difference between the third distance and the fourth distance.

In the noise removal process, whether the abnormal region is a fixed region or a variable region may be classified according to a difference value between the first distance and the second distance and a difference value between the third distance and the fourth distance.

In the fixed region, relative coordinates located in the vehicle do not change, and in the variable region, state coordinates located in the vehicle may change.

The noise removal process may be classified as a fixed area when the difference between the first distance and the second distance is less than a predetermined value, and when the difference between the third distance and the fourth distance is greater than or equal to a predetermined value. It can be classified as a variable region.

That is, the first and third abnormal regions may be classified as fixed regions, and the second and fourth abnormal regions may be classified as variable regions.

Denoising processing may be performed using a neural network model. The neural network model may include a convolutional neural network layer and may be trained to obtain an image from which the reflective area is removed by using training data including a plurality of vehicle images in which the reflective area is masked. The denoising process may include obtaining a vehicle image from which reflection areas are removed from target original images by using the learned neural network model.

On the other hand, in the above embodiments, it has been described based on the case of obtaining a denoised image based on three or two original images, but this is only an example, and the denoising process is performed on fewer or more original images. It may be performed based on.

Pre-processing may be performed on the acquired non-image data.

For example, the obtained non-image data may be transformed into a form that is easy to acquire vehicle information. Non-image data can be digitized. Non-image data may be converted into binary data form. Based on the plurality of non-image data, a score for obtaining vehicle information may be calculated.

2.3 Obtaining vehicle information

The vehicle management process may include obtaining vehicle information based on vehicle data.

The vehicle information may include parts information related to parts constituting the target vehicle, damage information related to internal/external damage of the target vehicle, vehicle model information related to the type of the target vehicle, and/or identification information for identifying the target vehicle. .

Vehicle information may be extracted from vehicle data or obtained by processing vehicle data. Vehicle information may be obtained using an algorithm designed to obtain vehicle information or a neural network model trained to obtain vehicle information, with vehicle data as input data.

For vehicle information acquisition, a classifier algorithm that classifies or predicts an input image for one or more labels may be used. For classification or prediction, various types of algorithms may be used. For example, k-nearest neighbor, support vector machine, artificial neural network, decision tree, self-organizing map, logical Logistic regression or the like may be used.

Artificial neural networks may be classifiers, hybrid classifiers, ensemble classifiers, linear regression neural networks, and the like. An artificial neural network can be a supervised, unsupervised or reinforcement learned model.

10 is a diagram for explaining a neural network model for obtaining vehicle information according to an exemplary embodiment.

Referring to FIG. 10 , a neural network model according to an embodiment may be prepared in the form of a classifier. According to one embodiment, the neural network model includes an input layer (IL), a pooling layer (PL), a convolutional neural network layer (CL), a fully connected layer (FCL), a hidden layer (HL), an output layer (OL), and the like, , a feature vector can be obtained based on the input image. The neural network model may be provided in the form of a classifier that classifies an input image into one or more labels. Alternatively, the neural network model may be prepared in the form of a regression model. The neural network model may be prepared as a regression model that obtains a linear output value for specific vehicle information based on an image as an input.

The neural network model may acquire output information by using a vehicle image obtained by photographing a part of the vehicle as an input. The output information may indicate whether an input image includes a target object. For example, an output layer of a neural network model may include an output node to which a probability function is assigned. The output layer of the neural network model may include an output node to which a probability function indicating whether the target image includes the target object is assigned. The output layer may include output nodes to which one or more probability functions indicating whether an input image includes each target object are assigned to one or more target objects.

For vehicle information acquisition, a neural network model that acquires a segmentation map (or image) including segmentation of one or more objects included in the image based on an input image may be used. In order to obtain the segmentation map, various types of neural networks may be used. For example, Fully Convolutional Network (FCN), SegNet, UNet, UNet++, DeConvNet, and the like may be used.

11 is a diagram for explaining a neural network model for obtaining vehicle information according to an exemplary embodiment. Referring to FIG. 11 , a neural network model according to an embodiment may be an image segmentation neural network model. The neural network model consists of an input layer (IL), a pooling layer (PL) that performs downsampling, a convolutional neural network layer (CL) that acquires feature maps, a deconvolutional neural network layer (DCL) that acquires segmentation maps, and a pooling layer (PL) that performs upsampling. It may include an unpooling layer (UPL) and an output layer (OL).

The pooling layer may perform max pooling, min pooling, average pooling, and the like. An unpooling layer may correspond to a pooling layer. The deconvolution layer may generate an intermediate segmentation map. The unpooling layer may upsample the intermediate segmentation map. The unpooling layer can perform maximum value unpooling, nearest neighbor unpooling, and the like.

For obtaining vehicle information, a neural network model that detects one or more objects from an input image may be used. For object detection, various types of neural network models may be used. It may be performed using You Only Look Once (YOLO), R-CNN, fast-RNN, faster R-CNN, SAS deep learning, SPPnet, and the like.

12 is a diagram for explaining a neural network model for obtaining vehicle information according to an exemplary embodiment. According to an embodiment, a fast-RNN model may be used to acquire part information. For example, referring to FIG. 12 , the neural network model according to an embodiment includes an input layer (OL), a convolutional neural network layer (CNL), a feature map (FM), a region proposal neural network (RPN), and a region of interest (ROI). ; Reason of Interest) It may include a deep learning neural network (DNN) including a bounding box (RBB), an ROI pooling layer (RP), and a fully connected layer.

The region proposal neural network may obtain one or more ROIs (or object proposals) or bounding boxes of the ROIs based on the feature map of the image. The region proposal neural network may obtain a score for each ROI (or object proposal) or its bounding box. The score may represent the degree to which the target ROI matches the detection target object.

The ROI pooling layer may perform pooling on individual ROIs. The ROI pooling layer may perform pooling on feature maps corresponding to individual ROIs. The ROI pooling layer may perform size adjustment on individual ROIs. The ROI pooling layer may obtain a feature map of a predetermined size. The fully-connected neural network may use a value obtained by flattening a feature map obtained through an ROI pooling layer.

The neural network model may obtain an identifier of an object included in an individual ROI or a bounding box of an individual object through a fully connected neural network layer and a hidden layer and an output layer connected to the fully connected neural network layer.

Meanwhile, the neural network models described above are only examples, and acquisition of vehicle information described in this specification is not limited thereto. Acquisition of vehicle information may be performed using an algorithm or neural network model not described herein.

The neural network model may acquire vehicle information by using additional information as an additional input. The neural network model may obtain vehicle information based on the image of the vehicle and the additional input generated based on the additional information. Additional input may be obtained based on additional information such as vehicle type, model name, manufacturer, year, and/or color. The additional input may be a feature value obtained based on an input image or an additional image different from the input image.

13 is a diagram for illustrating a neural network model according to an embodiment. Referring to FIG. 13 , a convolutional neural network and a neural network including a fully connected layer, a hidden layer, and an output layer (eg, an image-based classifier neural network) may further use additional information to obtain vehicle information. For example, the classifier model includes a convolutional neural network and at least one fully connected layer, and the at least one fully connected layer corresponds to an additional input value (AI) obtained based on feature values and additional information obtained through the convolutional neural network. It may include one or more corresponding nodes. The classifier model may include at least one node having an additional input value (AI) as an input value.

Additional information may be similarly used in the case of the neural network model illustrated in FIG. 11 . For example, the region segmentation neural network model according to an embodiment includes an encoding part and a decoding part, and vehicle information may be obtained by further using an additional input generated based on additional information. A region segmentation neural network model may include one or more nodes that use additional inputs generated based on additional information. The region segmentation neural network model may include one or more layers using additional inputs generated based on additional information as parameters.

Additional information may be similarly used in the case of the neural network model illustrated in FIG. 12 . For example, a neural network model according to an embodiment includes an input layer, a convolutional neural network layer, a feature map, a region proposal neural network, a reason of interest (ROI) layer, an ROI pooling layer, and a fully connected neural network, The fully-connected neural network may include a layer including at least one node having an additional input generated based on additional information as an input value.

According to an embodiment, a vehicle management assistance device for obtaining vehicle information based on a vehicle image and additional information may be provided.

The vehicle management assistant device for obtaining vehicle information related to the exterior of the vehicle based on the vehicle image includes an image acquisition unit that acquires a vehicle image obtained by photographing the outside of the vehicle, and additional information acquisition that obtains additional information related to characteristics of the vehicle. and a processing unit that obtains vehicle information by using a neural network model learned to obtain vehicle information based on the unit and the vehicle image.

The neural network model is trained to obtain a first feature set based on the vehicle image and is trained to acquire a second feature set based on a first part and a first feature set including at least one convolutional neural network layer and additional information. It may include a second part.

A neural network model may be a classifier or segmentation model. The neural network model may be prepared in the form of a CNN, Fully Convolutional Network (FCN), SegNet, UNet, UNet++, or DeConvNet.

A neural network model may be trained to obtain a set of features based on vehicle images. A neural network model may be trained to obtain parts information or damage information of a vehicle based on vehicle images. The neural network model may be trained using training data including a vehicle image labeled with part information or a vehicle image labeled with damage information.

A neural network model can be trained to obtain damage information or parts information based on vehicle images and additional information. The neural network model may be learned based on training data including additional information and vehicle images associated with the additional information and labeled with damage information or parts information. Additional information may be image or non-image information. The additional information related to the characteristics of the vehicle may be determined based on at least one of a vehicle manufacturer, color information, year model, and vehicle model. Additional information related to characteristics of the vehicle may be obtained based on a vehicle interior image obtained by photographing the interior of the vehicle.

The processing unit obtains a target vehicle image acquired by photographing the outside of the target vehicle through an image acquisition unit, obtains target additional information related to characteristics of the target vehicle through an additional information acquisition unit, and uses a neural network model based on the target vehicle image. Thus, target vehicle information on the target may be obtained.

A neural network model can be trained to acquire damage information related to damage outside the vehicle. The first feature set may include at least one feature value representing a part, type, color, configuration, shape, and the like of the vehicle. The second feature set may include at least one feature value related to whether or not the exterior of the vehicle is damaged. The second feature set may include a feature value indicating whether or not the vehicle has external damage. The second set of features may include one or more feature values indicative of the type of external damage to the vehicle. A feature value constituting a feature set may be obtained as a value of a probability function.

The neural network model may be trained to acquire part information related to parts constituting the exterior of the vehicle. The first feature set may include at least one feature value representing a part, type, color, configuration, shape, and the like of the vehicle. The second feature set may include at least one feature value related to a component constituting the exterior of the vehicle. The second feature set may include one or more feature values representing the types of parts constituting the vehicle. Each feature value may represent whether or not a specific part is included in the input vehicle image or a probability of including the specific part.

The second part may acquire a second feature set related to at least one part constituting the exterior of the vehicle, based on the first feature set and the additional information. The second feature set may include a first region feature set representing a region in which the first component is distributed and a second region feature set representing a region in which the second component is distributed.

The second part may obtain a second feature set related to at least one damaged region located outside the target vehicle, based on the first feature set and the additional information. The second feature set may include a damage region feature set indicating the location of the damaged region.

According to another embodiment, a vehicle management assistance method for obtaining vehicle information based on a vehicle image and additional information may be provided. Regarding the vehicle management assistance method, unless otherwise specified, the above-described vehicle management assistance device may be similarly applied.

30 is a diagram for explaining a vehicle management assistance method according to an exemplary embodiment.

Using a vehicle management assistance device including an image acquisition unit that acquires a vehicle image, an additional information acquisition unit that acquires additional information related to characteristics of the vehicle, and a processing unit that obtains vehicle information based on the vehicle image and the additional information, A vehicle management assistance method for obtaining information related to the exterior of a vehicle based on an image may be provided.

Referring to FIG. 30 , the vehicle management assistance method includes acquiring a target vehicle image (S3010), acquiring target additional information related to characteristics of the target vehicle (S3030), and targeting the target vehicle based on the target vehicle image and target additional information. Acquiring vehicle information (S3050) may be further included.

The vehicle management assistance method may include acquiring a target vehicle image for the target vehicle through an image acquisition unit (S3010). The target vehicle image may be an image of a vehicle photographed from an arbitrary direction. Alternatively, the target vehicle image may be an image taken from the front, side, or rear of the vehicle.

The vehicle management assistance method may include acquiring target additional information related to characteristics of the target vehicle through the additional information acquisition unit (S3030).

The vehicle management assistance method may include obtaining, by a processing unit, target vehicle information on the target vehicle based on the target additional information and the target vehicle image (S3050).

Obtaining vehicle information about the target vehicle (S3050) includes obtaining target vehicle information about the target vehicle by using a neural network model trained to obtain vehicle information based on the vehicle image based on the target vehicle image. may include more.

The neural network model is trained to obtain a first feature set based on the vehicle image and is trained to acquire a second feature set based on a first part and a first feature set including at least one convolutional neural network layer and additional information. It may include a second part.

The neural network model is trained to acquire damage information related to damage to the exterior of the vehicle, and the second feature set may include at least one feature value related to whether or not the exterior of the vehicle is damaged.

The neural network model is trained to acquire part information related to a part constituting the exterior of the vehicle, and the second feature set may include at least one feature value related to the part constituting the exterior of the vehicle.

The additional information related to the characteristics of the vehicle may be determined based on at least one of a vehicle manufacturer, color information, year model, and vehicle model.

Additional information related to characteristics of the vehicle may be obtained based on an image of the inside of the vehicle obtained by photographing the inside of the vehicle.

The second part may acquire a second feature set related to at least one part constituting the exterior of the target vehicle based on the first feature set and the additional information. The second feature set may include a first region feature set representing a region in which the first component is distributed and a second region feature set representing a region in which the second component is distributed.

The second part may obtain a second feature set related to at least one damaged region located outside the target vehicle, based on the first feature set and the additional information. The second feature set may include a damage region feature set indicating the location of the damaged region.

The vehicle management assistance methods described in this specification may be provided in the form of a computer readable recording medium storing a program for performing each method.

Meanwhile, the neural network model may be learned using training data. Learning data may be prepared differently according to the information to be acquired. The form of specific learning data and the learning of the neural network model will be described in more detail below in relation to the acquisition of each information.

Hereinafter, the type and acquisition method of each vehicle information will be described.

2.3.1 Obtaining Part Information

Part information related to parts constituting the vehicle may be obtained.

According to an embodiment, parts information may include the number, types, names, and identification information of parts constituting a vehicle.

Part information may be obtained based on non-image data. Part information may be determined by further considering vehicle information such as vehicle type, model name, manufacturer, year, and/or color. Parts information may be obtained from a part information database stored in advance based on vehicle information.

Part information may be obtained based on image data. Part information may be obtained from a pre-stored part information database based on vehicle information (eg, vehicle type, model name, manufacturer, color, etc.) obtained based on image data. Part information may be obtained from a pre-stored part information database based on image data and vehicle information acquired using a neural network model.

Part information may be obtained based on a parts information database in which the number, type, name, and identification information of parts are matched with classification information such as vehicle type, model name, manufacturer, and year. For example, the parts information database may match and store the number, type, name, and identification information corresponding to each part constituting a vehicle of a particular model year of a particular model with a corresponding year of the corresponding model.

According to another embodiment, the part information may include part area information included in the vehicle image and corresponding to each part. The parts area information may include boundaries of individual parts areas included in the vehicle image. The part area information may include a part area included in the vehicle image and a name or identification number of a part corresponding to each part area.

The part information may include masked part area information in a different format for areas corresponding to different parts included in the vehicle image. The part information may include part area information masked with different colors for areas corresponding to different parts included in the vehicle image or part area information labeled with different tags in areas corresponding to different parts.

14 is a diagram for explaining part area information according to an exemplary embodiment. Referring to FIG. 14 , part area information may include part area information corresponding to each part in a vehicle image.

Referring to (a) of FIG. 14 , parts information includes headlight area information (HL) corresponding to the headlight of the vehicle included in an image captured from the left front of the vehicle, and front bumper area information corresponding to the front bumper of the vehicle. (FB), left front wheel area information (LFF) corresponding to the left front fender of the vehicle, left front door area information (LFD) corresponding to the left front door of the vehicle, left rear door area corresponding to the left rear door of the vehicle Information (LRD), license plate area information (NP) corresponding to the license plate of the vehicle, left rear fender area information (LRF) corresponding to the rear fender of the vehicle, hood area information (HO) corresponding to the hood of the vehicle, etc. can Each region information may be associated with a name or identification information of a part corresponding to each region and stored.

Referring to (b) of FIG. 14 , part area information may include part area information corresponding to each part in a vehicle image captured from the front of the vehicle. Referring to (b) of FIG. 14 , part information includes headlight area information (LHL, RHL) corresponding to the headlights of the vehicle included in the front image of the vehicle, and front bumper area information (FB) corresponding to the front bumper of the vehicle. ), license plate area information (NP) corresponding to the license plate of the vehicle, side mirror area information (LSM, RSM) corresponding to the side mirror of the vehicle, windshield area information (FG) corresponding to the vehicle's windshield, vehicle emblem Emblem area information (EM) corresponding to , hood area information (HO) corresponding to the hood of the vehicle, and the like may be included. Each region information may be associated with a name or identification information of a part corresponding to each region and stored.

In FIG. 14 , parts area information has been described based on images captured from the left front and front sides of the vehicle, but this is only an example, and parts area information may be obtained for vehicle images captured from various directions.

According to one embodiment, part information may be obtained using a neural network model. Part information may be obtained using a neural network model that performs object recognition. Part information may be obtained using the neural network model described above with reference to FIGS. 10 to 13 .

The neural network model may acquire parts information by using a vehicle image as an input. The neural network model may obtain the type or name of a part included in the input vehicle image. The neural network model may acquire whether a specific part is included in the input vehicle image.

According to an embodiment, an output layer of the neural network model may include a probability function representing whether an input image includes a specific part. Also, for example, the output layer may include one or more probability functions indicating whether the input image includes each target part with respect to one or more target parts.

According to another embodiment, part information may be obtained using the neural network model described above with reference to FIG. 12 . Part information may include area information obtained through a neural network model and displaying areas corresponding to individual parts.

Referring to FIG. 15 , the neural network model according to an embodiment may obtain part information about parts included in a vehicle image by taking an image (a) of a part of a vehicle as an input. The neural network model may identify a part included in the input image by using the neural network model for obtaining part information, and obtain part information corresponding to the identified part. The neural network model may obtain part information indicating that the left front door is included in the input image by using a neural network model that obtains part information based on the image (a) in which a part of the vehicle is photographed.

Referring to FIG. 15 , the neural network model according to an embodiment may obtain part region information corresponding to a part included in a vehicle image by receiving an image (a) of a part of a vehicle as an input. The neural network model may use the neural network model to identify parts included in the vehicle image and obtain a region (or segmentation) corresponding to the identified parts. Referring to (b) of FIG. 16 , the neural network model may acquire area information corresponding to the left front door included in the vehicle image.

Referring to FIG. 16 , the neural network model according to an embodiment may obtain part information on one or more parts included in a vehicle image by taking an image (a) of a side of the vehicle as an input. The neural network model is based on the image (a) of a part of the vehicle and uses a neural network model that acquires parts information to indicate that the input image includes the left front door, left rear door, left front fender, and left rear fender. information can be obtained.

The neural network model may obtain a segmentation map in which regions corresponding to one or more parts are displayed by taking a vehicle image as an input. For example, the neural network model may acquire a segmentation map in which a first region corresponding to a first component and a second region corresponding to a second component are marked differently.

Referring to FIG. 16 , the neural network model may obtain part area information (b) corresponding to one or more parts included in the vehicle image by taking an image (a) of a side of the vehicle as an input. The neural network model may acquire an image (b) marked differently (eg, masked with different colors) with respect to a plurality of regions corresponding to individual parts included in the image. Referring to (b) of FIG. 16 , the neural network model may acquire area information corresponding to a left front door, a left rear door, a left front fender, and a left rear fender included in a vehicle image.

According to another embodiment, part information may be obtained using the neural network model described above with reference to FIGS. 10 to 13 . Part information is obtained through a neural network model and may include a bounding box displaying a region corresponding to an individual part.

The neural network model may obtain an identifier of an object included in an individual ROI or a bounding box of an individual ROI through a pre-connected layer and a hidden layer and an output layer connected to the fully-connected layer. The neural network model may obtain ROI bounding boxes for individual parts included in the vehicle image by taking the vehicle image as an input. The neural network model may obtain a classification result for an individual part region corresponding to an individual ROI included in the vehicle image by taking the vehicle image as an input.

A neural network model may be trained and prepared to acquire part information. The neural network model may be trained using part learning data including a plurality of vehicle images labeled with part information.

Part learning data may include a plurality of vehicle images. Part learning data may include a plurality of vehicle images photographed from different directions with respect to the vehicle. Part learning data may include a plurality of vehicle images photographed in the same direction with respect to the vehicle. Part learning data may include a plurality of vehicle images obtained by photographing vehicles of various types, models, or manufacturers. Part learning data may include a plurality of vehicle images obtained by photographing a vehicle of a single type, model, or manufacturer.

The part learning data may include a vehicle image to which part labels respectively corresponding to one or more parts included in the image are assigned. Part learning data may include vehicle images obtained by photographing various types of vehicles and having part labels attached thereto. Part labels may be assigned differently for each vehicle type, model, or manufacturer.

The component learning data may include at least one masked (or labeled) vehicle image with different formats for different components. The vehicle image may include at least one vehicle image masked (or labeled) with different colors for different parts or vehicle images labeled with different tags for regions corresponding to different parts.

Part learning data may include masked vehicle images for various parts. The part learning data may include one or more part areas, eg, left front door area information corresponding to the left front door and/or front bumper area information FB corresponding to the front bumper of the vehicle. The part learning data may include a plurality of vehicle images that include one or more pieces of common part area information and are photographed from different directions with respect to the vehicle. The part learning data may include one or more vehicle images including at least some different part area information. The part learning data may include one or more vehicle images masked with corresponding part information in all parts corresponding regions included in the image.

17 is a diagram for illustrating part learning data according to an embodiment. Referring to FIG. 17 , part learning data may include a plurality of vehicle images acquired by photographing vehicles of various models and labeled with a region corresponding to a specific part, for example, a left front door. A plurality of vehicle images may be acquired by photographing different types of vehicles.

18 is a diagram for illustrating part learning data according to another embodiment. Referring to FIG. 18, part learning data is obtained by photographing vehicles of various models, and regions corresponding to a plurality of parts, for example, a left front door, a left rear door, a left front fender, and a left rear fender are labeled differently. may include a plurality of vehicle images.

In FIGS. 17 and 18, the case where the part learning data includes a plurality of vehicle images photographed in the same direction with respect to the vehicle has been described, but this is only an example and the part learning data may include vehicle images photographed in various directions. can

The neural network model may be supervised, unsupervised, or reinforced-learned to obtain part information based on vehicle images using part learning data. A neural network model may be trained using a backpropagation method.

The neural network model may be trained to classify the vehicle image according to the parts included in the vehicle image using the component learning data described above. The neural network model may be trained to classify vehicle images according to parts included in the vehicle image, using training data including vehicle images to which a label corresponding to a part included in the vehicle image is assigned.

The neural network model may be trained to obtain whether a vehicle image includes a specific part through parts learning data. The neural network model may be trained to acquire whether each part is included in a target image for a plurality of parts through part learning data.

The neural network model may be trained to obtain region information about a region in which a specific part is distributed in a vehicle image through part learning data. The neural network model learns to acquire region information about the region where a specific part is distributed in a vehicle image through part learning data including vehicle images in which the types of individual parts included in the vehicle image and regions corresponding to the individual parts are labeled. It can be.

The neural network model may be trained to obtain region information about a region in which a specific part is distributed in a vehicle image through part learning data. The neural network model may be trained to obtain area information about a region in which each part is distributed, for each of a plurality of parts included in the vehicle image, through part learning data.

The neural network model may be learned using a plurality of component learning data. The neural network model is trained by taking a first part learning data including a vehicle image learned by taking a vehicle of the first type (or a vehicle of the first manufacturer) and a vehicle of the second type (or a vehicle of the second manufacturer). It may be learned using the second component learning data including the vehicle image.

Meanwhile, a plurality of neural network models may be learned and used. The first neural network model may be learned based on the first component learning data, and the second neural network model may be learned based on the second component learning data.

A neural network model can be trained to obtain parts information based on vehicle images and additional information. The neural network model may be trained to obtain parts information by further using additional information such as vehicle type, model name, manufacturer, and color. The neural network model may be trained to obtain part information based on a first feature set obtained based on the vehicle image and a second feature set obtained based on the additional information.

A neural network model can be trained to obtain parts information based on vehicle images and other images. The neural network model may be trained to obtain part information based on a first feature set obtained based on a vehicle image and a second feature set obtained based on another image (eg, a background image obtained by photographing a background). there is.

2.3.2 Obtain damage information

Damage information related to damage to the exterior of the vehicle may be obtained.

The damage information may be information related to damage such as scratches, dents, dents, breakage, loss, cracks, or the like on the exterior of the vehicle. The damage information may include information about the type, location, degree, number, occurrence time, and the like of damage to the exterior of the vehicle.

The damage information may include damage area information included in the vehicle image and corresponding to a location where the damage occurs. The damaged area information may indicate one or more damaged areas included in the vehicle image. The damaged area information may include a mark for visually distinguishing an area included in the vehicle image and corresponding to the damaged area. The damaged area information may include a boundary of the damaged area. Damage area information may include a bounding box indicating a damaged area. The damaged area information may include coordinate information of a damaged part in the vehicle image. The damaged area information may include corner coordinate information of the bounding box within the vehicle image. The damaged area information may include location information or coordinate information of a damaged part of the vehicle. Damaged area information may include absolute positional information (eg, coordinate information) of a damaged part based on a specific part of the vehicle (eg, the center line of a license plate, an emblem, etc.).

The damage information may include damaged area information corresponding to a damaged area included in the vehicle image and information related to the damaged area and matched with the damaged area information (eg, time of occurrence of damage, type of damage, degree of damage, etc.) there is.

19 is a diagram for explaining damage information according to an exemplary embodiment. Referring to (a) of FIG. 19 , damage information may include information related to damage to the exterior of a vehicle included in a vehicle image. Damage information may be obtained in the form of a tag or metadata assigned to a vehicle image. Referring to (b) of FIG. 19 , damage information may include a mark for emphasizing a damaged part in a vehicle image. Referring to (c) of FIG. 19 , the damage information may include a mark indicating a region where damage is likely to be located in a vehicle image, for example, a heat map. Referring to (d) of FIG. 19 , damage information may include a bounding box for displaying an area where damage is located in a vehicle image.

Damage information may be obtained based on image data. Damage information may be obtained using a neural network model trained to obtain damage information based on vehicle images. Damage information may be obtained using the neural network model described above with reference to FIGS. 10 to 13 .

The neural network model may acquire damage information by using a vehicle image acquired by photographing a part of the vehicle as an input. The neural network model may obtain whether or not there are damaged parts in the photographed vehicle based on the input vehicle image. The neural network model may acquire whether or not a damaged part exists in a specific area of the photographed vehicle based on the input vehicle image. The output layer of the neural network model may include an output node that obtains a probability value indicating whether the vehicle image includes the damaged area. The neural network model may obtain whether or not a damaged area is included in the photographed vehicle for one or more types of damage based on the input vehicle image. The output layer of the neural network model may include a plurality of output nodes that obtain probability values representing whether or not the vehicle image includes damage regions corresponding to a plurality of damage types.

The neural network model may obtain a damaged area based on a vehicle image obtained by photographing a part of the vehicle. The neural network model may detect an area corresponding to external damage of the vehicle from the vehicle image. The neural network model may obtain a bounding box or segmentation for a region included in the vehicle image and corresponding to external damage of the vehicle. The neural network model may obtain segmentation of a plurality of damaged regions. The neural network model may obtain a heat map. As exemplified in (c) of FIG. 19 , the neural network model may acquire a salience map in the form of a heat map for displaying an area where damage is located.

Referring to FIG. 20 , the neural network model according to an embodiment may obtain information on a damaged area corresponding to a damaged area by using an image (a) of a part of a vehicle as an input. The damage area information representing the damaged area may include a bounding box B1 obtained through a neural network model. Referring to FIG. 20 , the neural network model according to an embodiment may acquire damaged area information B2 corresponding to the damaged area by taking an image (b) of one side of the vehicle as an input. The damaged area information indicating the damaged area may include a bounding box obtained through a neural network model. The damaged area information may further include a damage type corresponding to the damaged area corresponding to each bounding box. The damage type can be obtained by the classifier part of the neural network model.

A neural network model can be trained to obtain damage information. The neural network model may be learned using damage learning data including a plurality of vehicle images labeled with damage information.

Damage learning data may include a plurality of vehicle images. The damage learning data may include a plurality of vehicle images taken from different directions with respect to the vehicle. The damage learning data may include a plurality of vehicle images taken in the same direction with respect to the vehicle. The damage learning data may include a plurality of damaged vehicle images obtained by photographing damaged vehicles of various types, models, or manufacturers. The damage learning data may include a plurality of damaged vehicle images obtained by photographing a damaged vehicle of a single type, model, or manufacturer.

The damage learning data may include a vehicle image to which a damage label representing the presence or absence of damage is assigned. The damage learning data may include vehicle images acquired by photographing various types of vehicles and assigned damage labels. The damage label may be given differently according to the type of damage (eg, scratch, dent, breakage, etc.).

The damage learning data may include a vehicle image masked in the damage area. The damage learning data may include a vehicle image masked (or labeled) with respect to one or more damage areas. The damage learning data may include vehicle images in which a plurality of types of damage areas are masked (or labeled) differently. For example, the damage learning data includes a vehicle image displayed in a first color for a first type of damage (eg, scratch) and a second color for a second type of damage (eg, dent). can do.

21 is a diagram for illustrating impairment learning data. Referring to (a) of FIG. 21 , damage learning data may include a vehicle image in which a damaged part is masked (M1). Referring to (b) of FIG. 21 , the damage learning data is masked (M2) with a first color for a damaged portion where the first type of damage (eg, distortion) is located, and the second type of damage ( For example, a vehicle image masked (M3) with a second color with respect to a damaged portion where a scratch is located may be included.

In FIG. 21, the case where the impairment learning data includes a plurality of vehicle images photographed in the same direction with respect to the vehicle has been described, but this is only an example and the impairment learning data may include vehicle images photographed in various directions. .

The neural network model may be learned using the damage learning data described above. The neural network model may be supervised, unsupervised, or reinforced-learned to obtain damage information based on vehicle images using the damage learning data. A neural network model may be trained using a backpropagation method.

The neural network model may be trained to classify vehicle images according to whether or not they include a damaged part using damage learning data including vehicle images labeled with or without damage. A neural network model may be trained to classify vehicle images according to the type of damage included in the vehicle images. The neural network model may be trained to acquire whether a target image includes each type of damage with respect to a plurality of damages through damage learning data including vehicle images labeled with respect to the presence or absence of multiple types of damage.

The neural network model may be trained to obtain damage area information through damage learning data. The neural network model may be trained to detect a damaged area from a target image using damage training data including an image of a vehicle masked to the damaged area for one or more types of damage. The neural network model provides area information about the area and/or type of each damage for each of a plurality of damages included in the vehicle image through damage learning data including vehicle images labeled for a plurality of types of damage areas. can be learned to acquire.

The neural network model may be learned using a plurality of impairment learning data. The neural network model is trained by photographing a vehicle having a first type of damage (eg, scratch) and first damage learning data including a vehicle image learned by photographing a vehicle having a second kind of damage (eg, dent). It may be learned using the second damage learning data including the vehicle image.

Meanwhile, a plurality of neural network models may be learned and used. A first neural network model may be learned based on the first impairment learning data, and a second neural network model may be learned based on the second impairment learning data. The second impairment learning data may be at least partially different from the first impairment learning data.

A neural network model can be trained to obtain damage information based on vehicle images and additional information. The neural network model may be trained to obtain damage information by further using additional information such as vehicle type, model name, manufacturer, and color. The neural network model may be trained to obtain damage information based on a first feature set obtained based on the vehicle image and a second feature set obtained based on the additional information.

A neural network model can be trained to obtain damage information based on vehicle images and other images. The neural network model may be trained to obtain damage information based on a first feature set obtained based on a vehicle image and a second feature set obtained based on another image (eg, a background image obtained by photographing a background). there is.

According to one embodiment, damage information may be used along with parts information. In this regard, it will be described in more detail below.

2.3.3 Obtaining vehicle type information

Vehicle type information related to the type of vehicle may be obtained.

The vehicle type information may indicate a vehicle type such as a sedan, a coupe, a wagon, a convertible, a hatchback, an SUV, a van, a truck, or a bus.

Vehicle model information may be obtained based on non-image data. Vehicle type information may be obtained based on vehicle registration number, VIN, model name, and the like. Vehicle type information may be obtained from a pre-stored database.

Vehicle type information may be obtained using image data. Vehicle type information may be obtained using a neural network model trained to obtain vehicle type information based on vehicle images. Vehicle type information may be obtained using the neural network model described above with reference to FIG. 10 .

The neural network model may obtain vehicle type information based on the vehicle image. The neural network model may classify vehicle images with respect to a plurality of vehicle model labels. For example, the neural network model may include at least one convolutional neural network layer and an output layer having one or more output nodes. The neural network model may include an output node that obtains a feature set related to a vehicle model based on a vehicle image through a convolutional neural network layer and obtains a probability value for at least one vehicle model label based on the feature set.

A neural network model may be trained to acquire vehicle type information. The neural network model may be learned using vehicle model learning data including a plurality of vehicle images labeled with vehicle model information. The neural network model may be supervised, unsupervised, or reinforced-learned to obtain damage information based on a vehicle image using model learning data. A neural network model may be trained using a backpropagation method.

The vehicle type learning data may include a plurality of vehicle images. The vehicle type learning data may include a plurality of vehicle images photographed in different directions or in the same direction with respect to the vehicle. The vehicle model learning data includes an image obtained by photographing a first vehicle model of a first manufacturer and to which the first vehicle model label has been assigned, and an image obtained by photographing a first vehicle model of a second manufacturer and to which the first vehicle model label is attached. can do.

The vehicle type learning data may include a plurality of vehicle images obtained by photographing vehicles of various colors or manufacturers, respectively, for a plurality of vehicle types. For example, the vehicle model learning data may include a first image set including a plurality of images obtained by photographing vehicles belonging to first vehicle types (eg, SUVs) of various manufacturers and to which first vehicle model labels are assigned, and first image sets of various manufacturers. A second image set including a plurality of images obtained by photographing vehicles belonging to two vehicle types (eg, sedans) and to which a second vehicle model label is assigned may be included.

A neural network model may be trained to acquire whether a target image is an image of a specific vehicle model. The neural network model uses vehicle model training data including a first image to which a first label is assigned indicating that the image is related to the first vehicle type and a second image to which a second label is assigned indicating that the image is not related to the first vehicle type. It may be learned to classify vehicle images according to whether or not they include the first vehicle type.

The neural network model may be trained to classify vehicle images according to vehicle types using vehicle model learning data. The neural network model uses vehicle model learning data including a first image to which a first label is attached indicating an image related to a first vehicle type and a second image to which a second label is attached indicating an image related to a second vehicle type, It can be learned to classify images according to the relevant car type.

Multiple neural network models may be trained and used. The first neural network model may be learned based on first vehicle model training data related to the first vehicle type, and the second neural network model may be learned based on second vehicle model training data related to the second vehicle type. The second vehicle model learning data may be at least partially different from the first vehicle model learning data. The first vehicle type learning data may include a vehicle image labeled with the first vehicle type, and the second vehicle type learning data may include a vehicle image labeled with a second vehicle type different from the first vehicle type.

According to an embodiment, a vehicle photographing guide may be provided based on vehicle model information. In this regard, it will be described in more detail below.

2.3.4 Obtaining identification information

Vehicle identification information for identifying the vehicle may be obtained.

The vehicle identification information may be a vehicle registration number, VIN, and the like.

Vehicle identification information may be obtained based on non-image data. Vehicle identification information may be obtained through user input. Vehicle identification information may be obtained from a pre-stored database.

Vehicle identification information may be obtained based on image data. Vehicle identification information may be obtained based on a vehicle image including an area corresponding to a license plate of the vehicle. The vehicle identification information may be obtained from an image obtained by photographing the front or rear of the vehicle. Vehicle identification information may be obtained through an algorithm, program, or neural network model that obtains vehicle identification information.

Vehicle identification information may be obtained based on the preprocessed image. For example, the vehicle identification information may be obtained from an image acquired through preprocessing such as gray scale, boundary enhancement (e.g., thresholding), contouring, rotation, and the like. Vehicle identification information may be obtained based on an area included in the vehicle image and corresponding to the identification information.

An area corresponding to the identification information may be extracted from the vehicle image using a horizontal/vertical edge method. For example, an area corresponding to the identification information may be extracted by detecting horizontal and vertical edge images in a vehicle image and setting a rectangle appearing as an edge as a candidate area.

An area corresponding to the identification information may be obtained through a contrast change. For example, the area corresponding to the identification information uses the fact that the contrast value has a continuous contrast vector from negative to positive or positive to negative in the number or letter area of the license plate when a line is read in the horizontal direction of the vehicle image. It can be obtained by estimating that there is a license plate area where the contrast change occurs symmetrically, and extending the target area from the estimated coordinates.

An area corresponding to the identification information may be obtained using a template. For example, the area corresponding to the identification information may be obtained by applying a reference circular set to a vehicle image, that is, using a template that recognizes a character by matching an input image to a standard pattern.

An area corresponding to the identification information may be obtained using color information.

An area corresponding to the identification information may be obtained using Hough transform. For example, the region corresponding to the identification information may be extracted by detecting an edge from the vehicle image using a Sobel operator and then searching for a group of vertical straight lines and a group of horizontal straight lines using Hough transform.

The area corresponding to the identification information may be obtained using run-length. For example, for the region corresponding to the identification information, a binary image is obtained by using the gradient in the neighboring direction, and then projected at an arbitrary angle, theta, and a run-length method is applied to the projected straight line to find a candidate region. It can be obtained by detecting a position where the ratio of the two straight lines is 2:1.

Pre-processing may be performed on a region corresponding to the identification information. For example, processing such as rotation for tilt correction, affine transformation, linear regression, and the like can be performed. Alternatively, preprocessing for shading correction may be performed. Alternatively, preprocessing for noise removal may be performed.

Vehicle identification information may be obtained using an algorithm for recognizing license plates. Vehicle identification information may be obtained using an Optical Character Recognition (OCR) algorithm. Vehicle identification information may be obtained using an OCR algorithm implemented through Tesseract, OpenCV, EAST detector, Python, or the like.

Vehicle identification information may be obtained through a neural network model. The neural network model may obtain vehicle identification information based on the vehicle image. The neural network model may obtain vehicle identification information based on a vehicle front image obtained by photographing the front of the vehicle and/or a vehicle rear image obtained by photographing the rear of the vehicle. The neural network model may obtain vehicle identification information based on a front license plate area included in the vehicle front image and corresponding to the license plate of the vehicle or a rear license plate area included in the vehicle rear image and corresponding to the vehicle license plate. The neural network model may obtain vehicle identification information based on a VIN area included in the vehicle image and corresponding to the VIN.

The neural network model may include a convolutional neural network. The neural network model may include a recurrent neural network (RNN) and/or a long short term memory (LSTM). The neural network model may obtain a feature set from the vehicle image through a convolutional neural network. The neural network model may acquire vehicle identification information (eg, vehicle number or VIN) corresponding to a vehicle image based on a feature set through a long-term memory or a recurrent neural network.

A neural network model may be trained to obtain vehicle identification information. The neural network model may be trained to obtain vehicle identification information based on a vehicle image using identification learning data including vehicle identification information or a vehicle image labeled with character information constituting the vehicle identification information. The neural network model may be supervised, unsupervised, or reinforced-learned to obtain damage information based on vehicle images using identification learning data. A neural network model may be trained using a backpropagation method.

The identification information learning data may include vehicle images photographed in various environments and labeled with vehicle identification information. The identification information learning data may include a vehicle image photographed in an environment such as night, daytime, dawn, rain, fog, or snow and to which an identification information label is attached.

The identification information learning data may include a vehicle image in which an area corresponding to the identification information of the vehicle is displayed. The identification information learning data may include one or more images to which an identification information label is assigned and masked to a region corresponding to vehicle identification information.

Identification information learning data is an image of the front of a vehicle obtained by photographing the front of the vehicle and labeled with the identification number of the vehicle and/or an image of the front of the vehicle obtained by photographing the rear of the vehicle and labeled with the identification number of the vehicle. A back image may be included.

The identification information learning data may include an identification area image for an area corresponding to vehicle identification information. For example, the identification information learning data is a front license plate image obtained by photographing the front license plate of the vehicle and assigned an identification number label, a rear license plate image obtained by photographing the rear license plate of the vehicle and assigned an identification number label, and/or a VIN of the vehicle. It may include a VIN image obtained by photographing and assigned an identification number (VIN).

According to one embodiment, driver identification information for identifying the driver (or owner) of the vehicle may be obtained. Driver identification information may be obtained based on non-image data such as user input or image data such as an image of a driver's identification card. When driver identification information is obtained based on image data, the above-described algorithm, program, or neural network model may be similarly used.

Vehicle identification information or driver identification information may be used for vehicle or driver history management, respectively. This will be described in more detail in the vehicle information management section.

2.3.5 Obtaining vehicle information Embodiment 1

According to an embodiment, part-damage information representing damage for each part may be obtained.

Referring to FIG. 22 , part-damage information is obtained by a vehicle image acquisition unit 221, a parts information acquisition unit 223, a damage information acquisition unit 225, and a part-damage information acquisition unit 227 including a device 220. ) can be obtained by Hereinafter, acquisition of part-damage information will be described with reference to FIG. 22 .

The vehicle image acquisition unit 221 may obtain a vehicle image acquired by photographing the exterior of the vehicle. The vehicle image obtaining unit 221 may obtain a vehicle image through a camera module or obtain a vehicle image from an external device.

The parts information acquisition unit 223 may acquire part information constituting the vehicle based on the vehicle image. The part information acquisition unit 223 may acquire the type (or identification information) of parts included in the input target vehicle image and part area information corresponding to each part. With regard to parts information, the description related to parts information acquisition may be applied.

The damage information acquisition unit 225 may obtain damage information related to damage to the exterior of the vehicle based on the vehicle image. The damage information acquisition unit 225 may obtain information on whether a damaged part exists in the input target vehicle image and/or damaged area information corresponding to the damaged part from the target vehicle image. Regarding damage information, the description related to acquisition of damage information may be applied.

The part-damage information acquisition unit 227 may obtain part-damage information indicating damage information for each individual part included in the target image based on the part information and damage information. The component-damage information may include whether or not there is a damaged part for each individual part included in the target image, the number of damaged parts, and/or damaged area information indicating the damaged part.

For example, the part-damage information acquisition unit 227 acquires part information and damage information, and determines whether there is a damaged area overlapping a part area corresponding to an individual part based on the part area information and the damaged area information. can The part-damage information acquisition unit 227 acquires part-damage information indicating that there is damage to the part (or indicating the number of damages) when there is a damage area overlapping a part area corresponding to a specific part. can do. The part-damage information acquisition unit 227 may acquire part-damage area information indicating the location of a damaged part existing in the part when there is a damaged area overlapping a part area corresponding to a specific part.

As a specific example, when the input target vehicle image is related to a first part and a second part, the first part corresponds to the first area of the target vehicle image and the second part corresponds to the second area of the target vehicle image, The part information may include first part identification information, second part identification information, a first area corresponding to the first part identification information, and a second area corresponding to the second part identification information. When damage occurs to the first part of the target vehicle, the damage information indicates that the target vehicle image includes the damaged area and may include damaged area information indicating the location of the damaged area. At this time, the part-damage information acquisition unit 227 compares the damaged area information with the first area and the second area in the target vehicle image, and determines that damage exists in the first area where the damaged area overlaps more. can do. The part-damage information obtaining unit 227 may acquire part-damage information indicating that the first part is damaged.

23 is a diagram for explaining acquisition of vehicle information according to an exemplary embodiment. Referring to FIG. 23 , the parts information acquisition unit may acquire a segmentation image (b) including region information corresponding to each part based on the vehicle image (a). The damage information acquisition unit may acquire a bounding box BB indicating a region where damage is located or an image (c) including the bounding box BB, based on the same vehicle image (a). The part-damage information obtaining unit may obtain damage information for each part based on the location of the bounding box BB and the segmentation corresponding to each part. Referring to FIG. 23 , the parts-damage information acquisition unit acquires parts-damage information indicating that damage is located in the left front door and the left rear door of the vehicle based on the distribution coordinates and segmentation information of the bounding box BB. can do.

2.3.6 Vehicle Information Acquisition Embodiment 2

According to an embodiment, part-damage information representing damage for each part may be obtained based on the part information and the damage information.

Referring to FIG. 24 , parts-damage information is obtained by a vehicle image acquisition unit 241, a parts information acquisition unit 243, a damage information acquisition unit 245, and a parts-damage information acquisition unit 247 including a device 240. ) can be obtained by The contents described above may be similarly applied to the vehicle image acquisition unit 241, the parts information acquisition unit 243, and the damage information acquisition unit 245 unless otherwise specified. Hereinafter, acquisition of part-damage information will be described with reference to FIG. 24 .

The part information acquisition unit 243 may acquire a part region (eg, a segmentation or bounding box of the part region) corresponding to an individual part.

The damage information obtaining unit 245 determines whether damage is included in the target part area based on the part area corresponding to the individual part included in the vehicle image, damage information indicating the type and number of damage included in the target part area, and /or Information on a damaged region where damage is located among target part regions may be obtained. The damage information acquisition unit 245 may obtain damage information for each individual part by using a neural network model learned to acquire damage information for each individual part area.

The part-damage information acquisition unit 247 may acquire part-damage information according to a target image based on damage information related to damage for each individual part. The component-damage information may include whether or not there is a damaged part for each individual part included in the target image, the number of damaged parts, and/or damaged area information indicating the damaged part.

As a specific example, when the input target vehicle image is related to a first part and a second part, the first part corresponds to the first area of the target vehicle image and the second part corresponds to the second area of the target vehicle image, The part information may include first part identification information, second part identification information, a first area corresponding to the first part identification information, and a second area corresponding to the second part identification information. When damage occurs to the first part of the target vehicle, the damage information obtaining unit 245 includes first damage information indicating that the damage is located in the first area and first damage information indicating the location of the damaged area included in the first area. 1 Damage area information can be obtained. The damage information obtaining unit may acquire damage information indicating that there is no damage to the second component. The part-damage information acquisition unit 247 may acquire part-damage information for the first part and the second part based on the damage information and the damage region information.

2.3.7 Obtaining Vehicle Information Embodiment 3

According to an embodiment, part-damage information representing damage for each part may be obtained based on the part information and the damage information.

Referring to FIG. 25, parts-damage information is obtained by a vehicle image acquisition unit 251, a vehicle model information acquisition unit 253, a parts information acquisition unit 255, a damage information acquisition unit 257, and a parts-damage information acquisition unit ( 259). The contents described above may be similarly applied to the vehicle image acquisition unit 251, the parts information acquisition unit 253, and the damage information acquisition unit 255 unless otherwise specified. Hereinafter, acquisition of part-damage information will be described with reference to FIG. 25 .

The vehicle model information acquisition unit 253 may obtain vehicle model information related to the vehicle model of the target vehicle based on the vehicle image. The vehicle type information obtaining unit 253 may be obtained through a user input or a pre-stored database.

The parts information acquisition unit 255 may obtain part information based on vehicle model information. The parts information obtaining unit may obtain part information by considering the vehicle model information and the vehicle image together. The part information acquisition unit 255 may obtain part information by using vehicle model information as an additional input and using a neural network model that obtains a feature set based on a vehicle image. The part information acquisition unit 255 may obtain part information and/or part area information by mapping a vehicle image and part area information using previously prepared part information and/or part area information for each vehicle type.

In general, since parts constituting a vehicle tend to be different for each vehicle type, accuracy may be improved when a part or a part region is determined in consideration of the vehicle model. Meanwhile, although vehicle type information is obtained based on a vehicle image, the vehicle type information may be obtained from a user input or pre-stored data.

The part-damage information acquisition unit 257 may acquire part-damage information based on the part information and the damage information. Regarding the acquisition of part-damage information, the above description with respect to FIGS. 22 and 23 may be similarly applied.

2.4 Vehicle information management

The vehicle management process may include managing vehicle information. Managing vehicle information may include managing the aforementioned parts information, damage information, vehicle type information, and/or identification information.

Managing vehicle information may include time-sequentially managing vehicle information. Managing vehicle information may include classifying vehicle information. Managing vehicle information may include managing one or more pieces of information in association with each other.

Hereinafter, management of vehicle information will be described with reference to some embodiments.

2.4.1 History management

Managing vehicle information may include history management of vehicle information. History management of vehicle information may mean time-sequential listing and/or management of one or more pieces of vehicle information obtained at different times.

Managing vehicle information may include time-sequentially managing part information based on one or more parts information. Managing vehicle information may mean managing part information indicating replacement, defect, damage, etc. of the corresponding part for each individual part along with time information indicating a time when each part information was acquired. For example, managing vehicle information may include obtaining first information indicating that a first part is damaged at a first time point, obtaining second information indicating that the first part has been replaced at a second time point after the first time point, and It may include managing first information and second information in chronological order. The first information or the second information may be obtained based on the vehicle image.

Managing vehicle information may include time-sequentially managing damage information (or part-damage information) based on one or more pieces of damage information. The vehicle management process may mean managing damage information representing damage to a specific vehicle together with time information representing a time when the damage information was acquired. For example, managing vehicle information acquires first information indicating that a first damage has occurred to a first vehicle at a first time point, and second damage has occurred to the first vehicle at a second time point after the first time point. It may include obtaining second information indicating, and managing the first information and the second information in chronological order. The first information or the second information may be obtained based on the vehicle image. The first information or the second information may include area information corresponding to each damage.

Managing vehicle information may include time-sequentially managing damage information based on one or more images. For example, managing vehicle information may include first damage information obtained based on a first vehicle image obtained at a first point in time and second damage information obtained based on a second vehicle image obtained at a second point in time after the first point in time. It may include managing damage information in consideration of time.

Managing vehicle information may be performed in consideration of user information. For example, managing vehicle information may include first damage information obtained based on a first vehicle image taken at a first time by a first user and a second vehicle image taken at a second time by a first user. It may include managing the second damage information obtained based on the time in consideration.

Managing vehicle information in consideration of user information is described in more detail in the following on-site vehicle management process and mobile-based vehicle management process.

2.4.2 Damage History Management Example 1 - New Damage

History management of damage information may include acquiring new damage information for newly occurring new damage. History management of damage information may include obtaining new damage information based on a plurality of images obtained by photographing the same vehicle at various times.

New damage information may be obtained based on a plurality of pieces of damage information. New damage information may be obtained based on first damage information corresponding to the first time point and second damage information corresponding to a second time point after the first time point.

The new damage information may include the number, degree, location (eg, coordinates) of newly generated damage.

The new damage information is based on first damage information acquired based on a first image taken at a first time point with respect to the first vehicle and a second image taken at a second time point after the first time point with respect to the first vehicle. It may be obtained based on the obtained second damage information. The new damage information may include information on newly occurring damage to the first vehicle after the first time point and before the second time point. For example, the new damage information may include the number (or degree) of damage according to the first damage information and the number of newly occurring damages obtained based on the number of damages according to the second damage information. The first image and the second image may be images obtained by photographing the vehicle in the same (or similar) direction.

The new damage information may include a location, size, distribution, and the like of a region corresponding to the new damage in the vehicle image.

The new damage information is based on first damage area information acquired based on a first image taken at a first time point with respect to the first vehicle and a second image taken at a second time point subsequent to the first time point with respect to the first vehicle. It may be obtained based on the second damaged area information obtained by doing. Based on the first damage area information and the second damage area information, the new damage information is about the location, size, or distribution of a damage area newly generated in the first vehicle after the first time point and before the second time point in the vehicle image. information may be included.

New damage information may be obtained for each part. The new damage information may include the number, degree, location (eg, coordinates), and location, size, distribution, and the like of a corresponding area of the newly generated damage to an individual part. By independently managing new damage information for each part, state management for each individual part can be facilitated.

The new damage information is first damage information on a first part acquired based on a first image captured at a first time point with respect to the first vehicle and a second damage information taken at a second time point after the first time point with respect to the first vehicle. It may be obtained based on the second damage information for the first part obtained based on the 2 images. Based on the first damage information and the second damage information, the new damage information may represent the number, size, location, and the like of newly occurring damage to the first part after the first time point and before the second time point.

26 is a diagram for explaining new damage information according to an embodiment. Referring to FIG. 26 , new damage information may be obtained based on a first vehicle image (a) obtained at a first point of view and a second vehicle image (b) taken at a second point in time.

First damage information indicating that there is one damage to the vehicle may be obtained based on the first vehicle image, and second damage information indicating that there are two damage to the vehicle may be obtained based on the second vehicle image. there is. Based on the first damage information and the second damage information, new damage information indicating that one new damage (B2 of the rear fender) has occurred can be obtained.

First damage information including first damage area information B1 may be obtained based on the first vehicle image (a). Second damage information including first damage area information B1 and second damage area information B2 may be obtained based on the second vehicle image (b). Based on the first damage information and the second damage information, damage to the rear fender of the vehicle that occurs after the first time point and before the second time point may be determined as new damage. The new damage information may include second damage area information corresponding to the damage of the rear fender.

2.4.3 Damage History Management Example 2 - Damage Superposition

There may be cases in which the newly occurred damage overlaps with the existing damage at least partially. In this case, the number of damages may not change even though new damages have occurred. In particular, this problem may occur when a new damaged area is included in an existing damaged area or an existing damaged area is included in a new damaged area. Even in this case, a means for notifying the user that new damage has occurred is required.

27 is a diagram for explaining damage history management when new damage overlaps with existing damage. When new damage overlaps with existing damage, the damage history may be acquired using the location, size, coordinates, and the like of the damaged part.

Referring to FIG. 27 , managing the damage history includes acquiring a first vehicle image (a) photographed at a first point in time and including first damage area information (B1) based on the first vehicle image (a). It may include obtaining first damage information. Managing the damage history is to obtain a second vehicle image (b) photographed at a second point in time after the first point in time and to obtain a second vehicle image (b) based on the second damaged area information (B2). It may include obtaining damage information.

Referring to FIG. 27 , second damaged area information B2 may overlap first damaged area information B1. The first damaged area information B1 may be included in the second damaged area information B1. When the first damage area information B1 overlaps at least a portion of the second damage area information B1, the number of damages according to the first damage information and the number of damages according to the second damage information may be the same.

New damage information may be obtained based on a difference between the second damage area information B2 and the first damage area information B1. The new damage information may be obtained based on a difference between the size of the second damaged area (second bounding box) and the size of the first damaged area (first bounding box) according to the second damaged area information B2. New damage information may be obtained based on a difference between the position (eg, center point coordinates) of the second damaged area (second bounding box) and the position (eg, center point coordinates) of the first damaged area (first bounding box). can New damage information may be obtained based on a difference between coordinates of corners of the second damaged region (second bounding box) and coordinates of corners of the first damaged region (second bounding box).

2.5 Vehicle information output and vehicle information output interface

The vehicle management system may output vehicle information. Vehicle information may be output through a user terminal having a display, a manager device, and the like.

2.5.1 Output Information

The vehicle management system may output acquired vehicle information and/or auxiliary information obtained based on the vehicle information.

The vehicle management system may output vehicle information such as parts information, damage information, vehicle type information, and identification information. The vehicle management system may output user information and/or location information together.

The vehicle management system may output a vehicle image and vehicle information corresponding to the vehicle image. The vehicle management system may output a plurality of vehicle images and vehicle information corresponding to each vehicle image. The plurality of vehicle images may be images taken from various angles with respect to the vehicle.

The vehicle management system may output a vehicle image and a direction indicator indicating a photographing direction of the image of the vehicle. A photographing direction may be obtained based on the vehicle image. The direction in which the vehicle is photographed may be obtained through a user's input.

28 is a diagram for explaining an output screen according to an exemplary embodiment.

Referring to FIG. 28 , an output screen according to an embodiment includes a vehicle image display unit (CI), a direction indicator (DI) indicating a capturing direction of a vehicle image, a non-image information display unit (IF) related to the vehicle, and vehicle damage information. It may include a damage information table (DT) indicating.

The vehicle image display unit CI may display an image from which noise is removed. The vehicle image display unit CI may display damaged area information. The damaged area information may include an existing damaged area ED and a new damaged area ND. The existing damaged area ED and the new damaged area ND may be displayed differently. Referring to FIG. 28 , the existing damaged area ED and the new damaged area ND may be displayed as bounding boxes of different colors or shapes.

The direction indicator DI may indicate a photographing direction of a displayed vehicle image. Referring to FIG. 28 , the direction indicator DI may indicate the left direction of the vehicle, and the vehicle image display unit CI may display a vehicle image captured from the left side of the vehicle. Vehicle images photographed in different directions may correspond to each of a plurality of directions indicated by the direction indicator DI. A corresponding vehicle image may be displayed in response to a user input for selecting a direction of the direction indicator DI.

The non-image information display unit IF may display non-image information related to the vehicle. Referring to FIG. 28 , the non-image information display unit IF may display a time point at which a vehicle image was captured, a vehicle number, a vehicle VIN, a vehicle model, and the like.

The damage information table DT may indicate the number of damages for each part corresponding to the vehicle image. The part corresponding to the image may refer to a part corresponding to the direction of the currently displayed vehicle image. For example, since an image obtained by photographing the left side of the vehicle is displayed in FIG. 28 , the damage information table DT may indicate the number of damages for each part for the parts located on the left side of the vehicle. The number of damages per part may include the number of existing damages and the number of new damages (separated by +) per part. Referring to FIG. 28 , the damage information table DT may indicate that existing damage exists on the left front door, left door handle, and left rear fender, and new damage exists on the left front door and left rear fender.

Meanwhile, the damage information table DT is only one embodiment of the damage information display unit, and information on existing damage or new damage may be displayed in other forms.

According to one embodiment, the vehicle management system may output auxiliary information obtained based on vehicle information. Auxiliary information is obtained based on vehicle information and may be information for assisting vehicle management.

The auxiliary information may be determined according to the degree of damage according to the damage information. For example, the auxiliary information may instruct necessary measures such as vehicle repair, painting, and replacement of some parts according to the degree of damage.

The auxiliary information may include repair cost information. Repair cost information may be obtained based on damage information. The repair cost information may include an estimate of a cost required to properly repair the vehicle based on the damage information.

Auxiliary information may include insurance-related information. The auxiliary information may include insurance claim information determined based on pre-stored insurance data.

2.5.2 Vehicle information output Embodiment 1 - Selection of output image when using multiple images

According to an embodiment, a plurality of vehicle images may be used to obtain vehicle information. For example, as described above in 2.2.3, there may be a case in which a plurality of images are used to remove noise of a vehicle image. In the case of outputting vehicle information obtained by using a plurality of images as described above, it will be examined which image among the plurality of images is used as a reference to output the vehicle information.

When vehicle damage information (or part-damage information) is obtained using a plurality of images, the vehicle image and the acquired damage information, particularly damage area information, may be overlapped and provided to the user. In this case, the vehicle image on which the damaged area information is overlapped may be any one of a plurality of images. The vehicle image on which the damaged area information is overlapped may be an image generated based on a plurality of images.

29 is a diagram for explaining display of damaged area information according to an exemplary embodiment. Referring to (a), (b) and (c) of FIG. 29 , processing for removing noise such as a background may be performed in the same manner as described in FIG. 7 . Referring to (d) of FIG. 29 , a vehicle image in which a bounding box BB corresponding to a damaged part of the vehicle is overlapped may be provided. The vehicle image overlapping the bounding boxes BB may be a vehicle image from which a background is removed, which is generated based on a plurality of vehicle images (a), (b), and (c). Alternatively, the vehicle image overlapping the bounding boxes BB may be an image in which the location of the vehicle is close to the center of the frame among the plurality of vehicle images (a), (b), and (c). Alternatively, the vehicle image overlapping the bounding boxes BB may be an image having the highest vehicle display ratio among the plurality of vehicle images (a), (b), and (c). The display ratio of the vehicle may refer to the ratio of the area displayed on the image to the total outer surface of the vehicle.

In FIG. 29 , a case in which noise removal processing for background removal is performed based on a plurality of images has been described, but this may be similarly applied to a case in which noise removal processing for removing reflected light in a vehicle is performed. In the case of noise removal processing for removing reflected light, damage area information may be superimposed on an image from which reflected light is removed and then output.

3. On-site fleet management system

3.1 On-site system

According to one embodiment, a vehicle management system for managing a state of a vehicle using an image of a vehicle captured on-site may be provided.

34 is a diagram for explaining the configuration of an on-site vehicle management system.

Referring to FIG. 34 , the on-site vehicle management system according to an embodiment may include a management device 341 , a camera module 343 and a detection unit 345 .

The management device 341 is connected to the camera module 343 and the sensing means 345, and can control the camera module 343 and the sensing means 345. The management device 341 may receive data from the camera module 343 and the sensing unit 345 . The management device 341 may include at least one of a display, a user interface, a communication module, a memory, and a controller.

The management device 341 may be composed of at least one or more electronic devices. The management device 341 may be implemented in the form of a tablet, smart phone, or notebook.

Although not shown, the management device 341 may be connected to a server. When the management device 341 is connected to the server, the management device 341 may exchange data with the server. In this case, the server may be connected to the plurality of management devices 341 and receive and process data from the plurality of management devices 341 .

Alternatively, the management device 341 may serve as a server. In this case, the management device 341 may be connected to other management devices 341 or other electronic devices to exchange and process data.

The camera module 343 may capture a vehicle entering or exiting. The camera module 343 may transfer photographing data of a vehicle to the management device 341 .

The detecting means 345 may detect the position of a vehicle entering or exiting. The detecting unit 345 may detect a vehicle entering or leaving the vehicle. The detecting unit 345 may be an optical sensor or a circuit breaker (PB).

The sensing unit 345 transmits a sensing value to the management device 341, and the management device 341 may extract image data used for vehicle management based on the sensing value. The management device 341 may extract image data from among the captured data based on the sensing value.

An on-site vehicle management system may be used for fleet management.

An on-site fleet management system can be used for parking control. More specifically, the on-site vehicle management system may be used for a task in which a parking manager compares and analyzes an image of a vehicle entering a parking area and an image of a vehicle exiting the parking area.

Also, the on-site fleet management system can be used for taxi management tasks of taxi companies. More specifically, the on-site fleet management system can be used by taxi companies to compare and analyze vehicle images of drivers before and after driving taxis.

3.2 On-Site Fleet Management Process

According to one embodiment, a vehicle management process may be provided for managing the state of a vehicle using images taken of the vehicle on-site. The on-site vehicle management process may be performed by the aforementioned on-site vehicle management system.

35 is a diagram for explaining a series of steps performed by the on-site vehicle management process.

Referring to FIG. 35 , the on-site vehicle management process includes acquiring vehicle data related to the vehicle and photographed on-site (S3510), acquiring vehicle information based on the acquired vehicle data (S3530), and It may include managing information (S3550) and outputting vehicle information (S3570). Hereinafter, each step will be described in more detail.

3.2.1 On-Site Vehicle Data Acquisition

The on-site vehicle management process may include acquiring vehicle data required for vehicle management (S3510).

Vehicle data may be prepared in various forms. For example, vehicle data may be image data or non-image data. Vehicle data may be pre-stored in the device or obtained from the outside. For example, vehicle data may be obtained through the camera module 343 or through a user input module.

The acquisition of vehicle data and vehicle data will be described in more detail below with specific examples.

3.2.1.1 Image data acquisition

36 is a diagram for explaining a step (S3510) of acquiring vehicle data required for vehicle management in the on-site vehicle management process. Hereinafter, the step of obtaining photographed vehicle data (S3510) will be described in more detail with reference to FIG. 36 .

Referring to FIG. 36 , acquiring vehicle data (S3510) according to an embodiment includes acquiring image data including a plurality of images (S3511) and extracting valid image data from among the acquired plurality of images. (S3513) may be included.

According to an embodiment, the vehicle data may be image data including a vehicle image obtained by photographing the vehicle. The vehicle image may be an image including some parts of the vehicle. The vehicle image may be an image obtained by photographing a part of the vehicle to include identification information for identifying the vehicle.

The vehicle data may be vehicle image data obtained by photographing the vehicle. The vehicle data may include image data and image data extracted from the image data.

The image data may refer to data captured by the camera module 343 between a plurality of specific points in time. In this case, the plurality of specific viewpoints may mean different viewpoints. Hereinafter, the plurality of specific viewpoints will be described in more detail with reference to the drawings by way of specific examples.

FIG. 37 is a diagram for explaining that image data is captured by the camera module 343 at a specific point in time when image data is sensed by the sensing unit 345 according to an exemplary embodiment.

Referring to FIG. 37 , the vehicle management system according to an embodiment may include a management device 341 , a detection means 345 and a camera module 343 . In addition, the vehicle management system may further include a circuit breaker (PB) for controlling entry or exit of the vehicle.

The camera module 343 may include at least one camera. The camera module 343 may include a first camera C1 and a second camera C2.

The detection unit 345 may include at least one or more sensors. The detecting unit 345 may include a first sensor S1 and a second sensor S2.

The circuit breaker (PB) may be controlled by the management device 341. The management device 341 may control the breaker PB when a vehicle entering or exiting approaches the breaker PB. The management device 341 may control the barrier PB when a vehicle entering or exiting approaches the barrier PB and entering or exiting the vehicle is permitted. The management device 341 may control the circuit breaker PB based on a sensing value of at least one of the first sensor S1 and the second sensor S2. The management device 341 may control the circuit breaker PB when the vehicle is detected by the first sensor S1.

The camera module 343 may be installed in an area adjacent to a movement path of a vehicle entering or exiting a parking lot in order to photograph a vehicle entering or exiting a parking lot. Hereinafter, the camera module 343 for photographing the entering vehicle will be described.

The first camera C1 and the second camera C2 may be installed to photograph different sides of the vehicle. For example, the first camera C1 may be installed at a location capable of capturing a first side of the vehicle, and the second camera C2 may be installed at a location capable of capturing a second side of the vehicle. The first camera C1 and the second camera C2 may be installed facing each other.

The first camera C1 may be installed at a location capable of taking pictures of the side and rear of a vehicle entering or exiting. The first camera C1 may be installed so that the second camera C2 is located in a photographing area. The first camera C1 may be installed facing the second camera C2. The first camera C1 may additionally capture the back of the vehicle entering or exiting, so that non-image data may be additionally acquired.

The second camera C2 may be installed at a location capable of taking pictures of the side and front of a vehicle entering or exiting. The second camera C2 may be installed so that the first camera C1 is located in a photographing area. The second camera C2 may be installed facing the first camera C1. The second camera C2 may additionally capture the front of the vehicle entering or exiting the vehicle, so that non-image data may be additionally acquired.

The first camera C1 and the second camera C2 may be positioned parallel to the blocker PB.

Alternatively, the first camera C1 and the second camera C2 may be spaced apart from each other. The first camera C1 may be located in an area spaced apart from the circuit breaker PB, and the second camera C2 may be located in an area corresponding to the circuit breaker PB. Since the first camera C1 is located in an area separated from the blocker PB, it is possible to shorten the processing time by acquiring non-image information of the vehicle at a faster time point. The first sensor S1 may be located in an area spaced apart from the circuit breaker PB, and the second sensor S2 may be located in an area corresponding to the circuit breaker PB. The first sensor (S2) may be installed spaced apart from the breaker (PB) to detect a vehicle entering the parking lot, and the second sensor (S2) is the breaker (S2) to detect whether the vehicle is adjacent to the breaker (PB). PB) and can be installed to be located in the corresponding area.

The management device 341 may acquire photographing data from the first camera C1 and the second camera C2. The management device 341 may extract image data from the captured data based on sensing values transmitted from the first sensor S1 and the second sensor S2. Here, the image data may include data actually used for vehicle management.

Alternatively, the management device 341 detects at least one of the first camera C1 and the second camera C2 based on the sensing value transmitted to at least one sensor of the first sensor S1 and the second sensor S2. You can control one. For example, when the entry of the vehicle is recognized by the first sensor S1, the first camera C1 and the second camera C2 may be controlled to capture the vehicle, and the second When entering the vehicle is completed by the sensor S2, the first camera C1 and the second camera C2 may be controlled to change to a state in which the vehicle is not photographed.

The management device 341 may define photographing data between the time when the vehicle completely passes the first sensor S1 and the time when the vehicle is first sensed by the second sensor S2 as image data. In this case, a time point when the vehicle is recognized by the first sensor S1 and then recognition is terminated may be defined as a start point, and a point in time when the vehicle is recognized by the second sensor S2 may be defined as an end point. The management device 341 may define photographing data obtained between the start time and the end time as image data.

The management device 341 defines photographing data between the time when the vehicle completely passes the first sensor S1 and the time when the vehicle is first detected by the second sensor S2 as image data, and uses the image data. By using it as data for vehicle management, the amount of data stored in the storage unit of the management device 341 can be reduced, and the first camera C1 and the second camera C2 are transferred to the management device 341 This has the effect of reducing the amount of data.

The first camera C1 may be located in an area corresponding to the first sensor S1, and the second camera C2 may be located in an area corresponding to the second sensor S2. By locating the first camera C1 in an area corresponding to the first sensor S1 and defining a point in time when the vehicle completely passes the first sensor S1 as a starting point, the first camera C1 can detect the vehicle Data obtained by photographing all regions on one side of the image may be defined as image data. In addition, by locating the second camera C2 in an area corresponding to the second sensor S2 and defining the point at which the vehicle is initially detected by the second sensor S2 as the end point, the second camera C2 is Data obtained by photographing all areas of the other side of the vehicle may be defined as image data. Accordingly, the image data of the first camera C1 and the second camera C2 is defined as data capturing all areas on both sides of the vehicle, and accordingly, damage to the vehicle can be more accurately recognized.

38 is a diagram for explaining that image data is captured by the camera module 343 at a specific point in time when image data is sensed by a sensor according to another embodiment.

Referring to FIG. 38 , the management device 341 according to another embodiment captures photographed data between the time when the vehicle is first sensed by the first sensor S1 and the time when the vehicle is first sensed by the second sensor S2 as an image. data can be defined. In this case, the time point at which the vehicle is recognized by the first sensor S1 may be defined as the start time point, and the time point at which the vehicle is recognized by the second sensor S2 may be defined as the end time point. The management device 341 may define photographing data obtained between the start time and the end time as image data.

The management device 341 defines the photographing data between the time when the vehicle is first sensed by the first sensor S1 and the time when the vehicle is first sensed by the second sensor S2 as image data. can advance The management device 341 has the effect of enabling smoother vehicle management by accurately securing vehicle identification information such as vehicle number or VIN information located in the front of the vehicle by defining photographic data at an earlier point in time as image data. there is

39 is a diagram for explaining that image data is captured by the camera module 343 at a specific point in time when image data is sensed by a sensor according to another embodiment.

Referring to FIG. 39 , the management device 341 according to another embodiment captures photographed data between the time when the vehicle is initially detected by the first sensor S1 and the time when the vehicle completely passes the second sensor S2. It can be defined as video data. In this case, a time point at which the vehicle is recognized by the first sensor S1 may be defined as the start time point, and a time point at which the vehicle is recognized by the second sensor S2 and then recognition is terminated may be defined as the end time point. The management device 341 may define photographing data obtained between the start time and the end time as image data.

The management device 341 defines photographing data between the time when the vehicle is first sensed by the first sensor S1 and the time when the vehicle completely passes the second sensor S2 as image data, which is shown in FIGS. 37 and 38. It is possible to advance the start time and delay the end time. The management device 341 defines the image data taken at an earlier time point as image data, and defines the captured data at a later time point as image data. can be performed, and as a result, there is an effect of minimizing omission of damage information occurring on the exterior of the vehicle.

Meanwhile, although not shown in the drawings, the vehicle management system according to an embodiment may further include a third sensor, a fourth sensor, a third camera, and a fourth camera.

The above-described first sensor S1, second sensor S2, first camera C1, and second camera C2 may be used to secure image data when a vehicle enters a parking area, and the third sensor , The fourth sensor, the third camera, and the fourth camera may be for securing image data when the vehicle leaves the parking area.

In other words, the third sensor may further include a third sensor and a third camera when entering the vehicle, and the third sensor may play a role corresponding to the first sensor S1 when entering the vehicle. 1 It can play a role corresponding to the camera (C1). In this case, the third camera may be installed to photograph the same side of the vehicle that the first camera C1 photographs. In addition, the fourth sensor may further include a fourth sensor and a fourth camera when entering the vehicle, and the fourth sensor may play a role corresponding to the second sensor S2 when entering the vehicle. It may play a role corresponding to the camera C2. In this case, the fourth camera may be installed to photograph the same side of the vehicle as the second camera C2.

As described above, the first sensor S1, the second sensor S2, the first camera C1, and the second camera C2, the third sensor, the fourth sensor, the third camera, and the fourth camera Among the contents, overlapping contents with the first sensor S1, the second sensor S2, the first camera C1, and the second camera C2 are omitted.

40 is a diagram for explaining that image data is captured by the camera module 343 at a time within a certain range from the time when the circuit breaker PB operates according to another embodiment.

The vehicle management system according to another embodiment of FIG. 40 is the same as the embodiment of FIG. 37 except that the circuit breaker (PB) is used as the sensing means 345 compared to FIG. 33 . Therefore, in describing the vehicle management system according to the embodiment of FIG. 40, the same reference numerals are assigned to portions common to those of FIG. 37, and detailed descriptions thereof are omitted.

Referring to FIG. 40 , a vehicle management system according to another embodiment may include at least one circuit breaker (PB) and at least one camera module 343.

The camera module 343 may include a first camera C1 and a second camera C2. The first camera C1 may be positioned in an area spaced apart from the blocker PB, and the second camera C2 may be positioned in an area corresponding to the blocker PB. The first camera C1 and the second camera C2 may be installed to be spaced apart from each other, and the first camera C1 and the second camera C2 may photograph different sides of the vehicle.

The management device 341 may acquire photographing data from the first camera C1 and the second camera C2. The management device 341 may extract image data from photographing data acquired from the first camera C1 and the second camera C2. Here, the image data may include data actually used for vehicle management.

The image data may refer to images captured between a plurality of specific points in time among the captured data captured by the camera module 343 . In this case, the plurality of specific points in time may mean points within a certain range from the point in time when the circuit breaker PB operates. On the other hand, a point in time within a certain range from the point in time at which the circuit breaker PB is operated is a point in time that has a certain range so that at least some of the main parts of the entire exterior of the vehicle can be included in the image data when the vehicle is photographed at a point in time within the certain range. can mean That is, when a vehicle approaches the parking area, the circuit breaker PB operates, and a plurality of specific points in time include a first time point prior to a certain time point and a certain time point based on the time point at which the breaker PB operates according to the vehicle's approach. It may refer to a second point in time later.

In other words, the image data may refer to an image captured by the camera module 343 between a first point in time before a certain point in time and a second point in time after a certain point in time based on the point in time at which the circuit breaker PB operates.

The management device 341 may define photographing data between the first and second viewpoints of the vehicle as image data. In this case, the start point can be defined as a point before a certain point based on the point at which the circuit breaker (PB) operates, and the point in time when the vehicle completely passes the circuit breaker (PB) after the circuit breaker (PB) operates is defined as the end point. can The management device 341 may define photographing data obtained between the start time and the end time as image data.

The management device 341 defines the photographing data obtained between the start time and the end time as image data, and uses the image data as data for vehicle management, thereby blocking the circuit breaker even in a parking area without a sensor. There is an effect of enabling the vehicle management function to be performed using only (PB).

Also, the second time point may be a time point corresponding to the time point at which the circuit breaker PB operates. In this case, there is an effect of reducing the amount of data transmitted from the first camera C1 and the second camera C2 to the management device 341.

On the other hand, the camera module 343 according to an embodiment may be a fixed type in which a shooting point and an angle are fixed during shooting, but is not limited thereto. As another example, the camera module 343 according to an embodiment may be of a rotary type that rotates according to the movement of the vehicle during photographing.

When the camera module 343 according to an embodiment is provided in a rotatable manner, the above-described plurality of specific viewpoints may be variable within a range in which image data may include at least some of the main parts of the exterior of the vehicle.

Hereinafter, a method of extracting image data suitable for an on-site vehicle management process from the image data will be described with reference to drawings.

The image data extracted from the image data may be any one of vehicle images captured by the camera module 343 at a plurality of specific points in time. Also, the image data extracted from the image data may be any one of vehicle images captured by the camera module 343 between a plurality of specific points in time. The image data may be any one of vehicle images taken by the camera module 343 between the start time and the end time.

41 is a diagram for explaining a method of extracting image data from image data according to an exemplary embodiment.

As described above, image data acquired by the camera module 343 at a plurality of specific points in time may include a plurality of images. At this time, valid image data may be extracted from among the plurality of images in order to allow the on-site vehicle management process to be normally performed.

When photographing data acquired between the starting time and ending time by the management device 341 is defined as image data, valid image data may be at least one of image data between the starting time and ending time.

The effective image data may be at least one image in an intermediate section of image data. The middle section may be a specific section between a start time and an end time. The intermediate section may include an intermediate time point between the start time point and the end time point. In this case, the time interval between the start time and the middle time point and the time interval between the middle time point and the end time point may correspond.

The valid image data may refer to at least one or more image data including a main part of the exterior of the vehicle among a plurality of images included in the image data.

As a result, the management device 341 can reduce the amount of data stored in the storage unit of the management device 341 by using the effective image data as data for vehicle management, and effectively display major parts of the exterior of the vehicle. It has a manageable effect.

The image data obtained by the management device 341 may be a plurality of images including (a) to (c) images of FIG. 41 . The management device 341 can effectively manage the main parts of the exterior of the vehicle by extracting the image (b) showing the main parts as effective image data.

Meanwhile, the valid image data may be an image including only a partial area of the vehicle. Even in this case, the effective image data may be in the form of including the main parts of the vehicle's exterior, thereby having an effect of more clearly determining the vehicle's exterior.

3.2.1.2 Non-Image Data Acquisition

According to one embodiment, vehicle data in the form of non-images other than images may be obtained. Vehicle data may be non-image data. Vehicle data may be non-image data including vehicle-related information. The non-image data may be stored or acquired in advance. The non-image data may be acquired by the camera module 343 . The non-image data may be obtained based on photographing data captured by the camera module 343 . The management device 341 may obtain non-image data based on the image data.

3.2.1.3 Data Processing

Processing may be performed on the acquired data in the on-site fleet management process.

In the on-site vehicle management process, pre-processing (or post-processing) may be performed on the acquired image data.

The management device 341 may perform pre-processing on at least one effective image data among image data.

The management device 341 may perform noise removal processing based on a plurality of images of an intermediate section between a start time and an end time. The management device 341 may perform noise removal processing based on a plurality of valid image data.

42 is a diagram for illustrating noise removal processing using a plurality of images. Referring to FIG. 42 , the management device 341 obtains a plurality of valid image data (a) to (c) having a vehicle area and a background area, and based on the plurality of valid image data (a) to (c) A noise removal process may be performed to obtain an image (d) from which the background area has been removed. Based on the plurality of valid image data (a) to (c), the management device 341 removes an area corresponding to a background (i.e., background area), which is an object whose position is fixed in the effective image data, An image d including an area corresponding to a vehicle, which is an object whose location is changed within valid image data (ie, a vehicle area), may be acquired.

For example, in the case of a parking control company or a taxi company, since the camera module 343 photographs a moving vehicle at a fixed location, effective image data may include a moving vehicle image and a fixed background image. In this case, the background image may be repeatedly learned as a non-vehicle image by the neural network model. Accordingly, the denoising process may include obtaining a vehicle image from which the background region is removed from the effective image data by using a neural network model in which information on the background image is repeatedly learned.

The management device 341 may perform noise removal processing for removing reflected light or the like based on a plurality of valid image data. The management device 341 obtains a plurality of valid image data, and an object whose position is changed (eg, a reflective area corresponding to reflected light) and an object whose position is not changed (eg, a vehicle area or damage corresponding to damage to a vehicle) region), an object whose location is changed or an object whose location is not changed may be removed. For example, the management device 341 obtains a vehicle image from which the reflective region is removed, based on valid image data including a damaged region corresponding to a damaged part of the vehicle and a reflective region corresponding to light reflected by the surface of the vehicle. A removal process may be performed.

43 is a diagram for illustrating noise removal processing using a plurality of images. Referring to FIG. 43, the management device 341 obtains a plurality of valid image data (a) and (b) including a damaged area and a reflective area, and removes noise to obtain an image (c) from which the reflective area is removed. processing can be performed. Referring to FIG. 43 , the management device 341 is based on a plurality of valid image data (a) and (b), the image (c) from which the reflective area whose position or state is changed within the image or vehicle area is removed (c ) may be performed to obtain noise removal processing.

The image (c) from which the reflection area is removed may be generated based on any one of a plurality of effective image data (a) and (b). For example, the management device 341 obtains a reflection area based on a plurality of valid image data (a) and (b), removes the reflection area from any one of the plurality of effective image data (a) and (b), , a noise-removed image can be obtained.

The management device 341 may perform noise removal processing using a neural network model. The neural network model may include a convolutional neural network layer and may be trained to obtain an image from which the reflective area is removed by using training data including a plurality of vehicle images in which the reflective area is masked. The management device 341 may perform noise removal processing to obtain a vehicle image from which reflection areas are removed from valid image data using the learned neural network model.

An image on which the above-described noise removal process is performed may be defined as a damaged image. The management device 341 may determine vehicle information including damage to the vehicle based on the damage image.

When the camera module 343 is installed in a noise-free environment, the management device 341 may omit the noise removal process. In this case, acquisition of vehicle information to be described later may be performed based on valid image data. The damaged image at this time may be the valid image data.

3.2.2 On-site vehicle information acquisition

The on-site fleet management process may include obtaining vehicle information based on vehicle data.

The vehicle information may include parts information related to parts constituting the target vehicle, damage information related to internal/external damage of the target vehicle, vehicle model information related to the type of the target vehicle, and/or identification information for identifying the target vehicle. .

Vehicle information may be extracted from vehicle data or obtained by processing vehicle data. Vehicle information may be obtained using an algorithm designed to obtain vehicle information or a neural network model trained to obtain vehicle information, with vehicle data as input data.

Acquisition of vehicle information based on vehicle data has been described above in 2.3, so duplicated content is omitted.

According to an embodiment, the management device 341 may obtain vehicle information based on the damaged image. The management device 341 may obtain damage information based on the damage image.

According to an embodiment, in the case of a parking control company, the management device 341 may obtain parking vehicle information using a pre-stored database. The parked vehicle information may include vehicle type information indicating a model, manufacturer, model name, and year of the parked vehicle. In addition, the parked vehicle information may further include parts information indicating the number, type, name, identification information, and the like of parts constituting the parked vehicle.

The parked vehicle information may include additional damage information generated inside/outside the target vehicle. The additional damage information may be calculated based on a damage image of an incoming vehicle and a damage image of an outgoing vehicle. The additional damage information may be calculated by comparing a damage image of an entering vehicle with a damage image of an exiting vehicle.

The additional damage information may be calculated based on damage information calculated based on a damage image of an incoming vehicle and damage information calculated based on a damage image of an outgoing vehicle.

The management device 341 stores at least one of a damage image and damage information of an incoming vehicle, retrieves it when exiting the vehicle, and calculates additional damage information by comparing it with at least one of the damage image and damage information of the vehicle entering the vehicle. can do.

The management device 341 may store non-image data of the vehicle entering the vehicle in conjunction with the damaged image and damage information. For example, the management device 341 may store a damage image and damage information of a vehicle entered in association with the license plate number of the vehicle.

The management device 341 loads the damage image and damage information linked to the license plate number of the corresponding vehicle based on the recognized license plate number when the corresponding vehicle leaves the vehicle, and compares it with the damage image and damage information obtained when leaving the vehicle. Additional damage information can be calculated.

As another example, the management device 341 may further include additional information including an accident history of the vehicle when the parked vehicle uses the parking facility twice or more.

According to another embodiment, in the case of a taxi company, the management device 341 may obtain taxi vehicle information using previously stored data. The taxi vehicle information may include vehicle type information indicating the vehicle type, manufacturer, model name, year, manufacturing time, manufacturing place, manufacturer, location, and registration base location of the taxi vehicle. In addition, the taxi vehicle information may further include parts information indicating the number, type, name, and identification information of parts constituting the taxi vehicle.

Taxi vehicle information may include additional damage information generated inside/outside the target vehicle. The additional damage information may be calculated based on a vehicle damage image obtained before taxi operation and a vehicle damage image obtained after taxi operation. The additional damage information may be calculated by comparing a damage image of a vehicle acquired before driving a taxi with a damage image of a vehicle acquired after driving a taxi.

The additional damage information may be calculated based on damage information calculated based on a vehicle damage image obtained before taxi operation and damage information calculated based on a vehicle damage image obtained after taxi operation.

The management device 341 stores at least one of a vehicle damage image and damage information obtained before driving a taxi, and retrieves it after driving a taxi to store at least one of a damage image and damage information of a vehicle obtained after driving a taxi. Comparison can yield additional damage information.

The management device 341 may store non-image data of the vehicle acquired before driving the taxi in conjunction with the damage image and damage information. For example, the management device 341 may store the damage image and damage information of the vehicle obtained before driving the taxi in association with the license plate number of the vehicle.

The management device 341 loads the damage image and damage information linked to the license plate number of the corresponding vehicle based on the recognized license plate number when the corresponding vehicle completes the taxi operation, and loads the damage image and damage information obtained after driving the taxi. Additional damage information can be calculated by comparing with .

3.2.3 Vehicle information management

The on-site vehicle management process may include managing vehicle information. Managing vehicle information may include managing the aforementioned parts information, damage information, vehicle type information, and/or identification information.

Also, managing the vehicle information may include managing the vehicle information time-sequentially. Managing vehicle information may include classifying vehicle information. Managing vehicle information may include managing one or more pieces of information in association with each other.

Hereinafter, management of vehicle information will be described with reference to some embodiments.

3.2.3.1 History management

Managing vehicle information may include history management of vehicle information. History management of vehicle information may mean time-sequential listing and/or management of one or more pieces of vehicle information obtained at different times.

According to an embodiment, managing vehicle information in an on-site vehicle management process may mean managing damage information representing damage to a specific vehicle together with time information representing a time when the damage information was acquired. can For example, managing vehicle information acquires first information indicating that a first damage has occurred to a first vehicle at a first time point, and second damage has occurred to the first vehicle at a second time point after the first time point. It may include obtaining second information indicating, and managing the first information and the second information in chronological order.

Illustratively, managing vehicle information in a parking management process may mean managing damage information representing damage to a parked vehicle together with time information representing a time when the damage information was acquired. For example, managing vehicle information in a parking management process means obtaining first information indicating that a first damage has occurred to the first parked vehicle when the first parked vehicle enters the parking area, and the first parking vehicle is in the parking area. It may include acquiring second information indicating that the second damage occurs in the first parked vehicle when the vehicle is taken out, and managing the first information and the second information in chronological order.

As another example, managing vehicle information in a taxi vehicle management process may mean managing damage information representing damage to a taxi vehicle together with time information representing a time when the damage information was obtained. For example, managing vehicle information in a taxi vehicle management process means obtaining first information indicating that a first damage has occurred to the first taxi before the first taxi runs, and after the first taxi runs, a second damage to the first taxi is obtained. It may include acquiring second information indicating that damage has occurred, and managing the first information and the second information in chronological order.

3.2.3.2 Damage history management Example 1 - Management by driver

A history of damage information can be managed in an on-site fleet management process. History management of damage information may include acquiring new damage information for newly occurring new damage. History management of damage information may include obtaining new damage information based on a plurality of images obtained by photographing the same vehicle at various times.

The damage information history may be managed for each driver in consideration of driver information.

According to an embodiment, the management device 341 may manage damage information records of taxi vehicles for each driver in consideration of driver information. The management device 341 manages the damage information history of the taxi vehicle for each driver, so that when damage occurs to the taxi vehicle, the responsibility can be clarified, and the current or past damage to the taxi vehicle is caused by a certain driver. It is easy to determine whether or not it is, and it can provide an effect of reducing the cost of taxi vehicle management accordingly. In addition, the management device 341 manages the damage information history of the taxi vehicle for each driver, so that the driver can be evaluated, for example, safe driving score, etc. It is possible to provide an effect of facilitating management of drivers in the company by issuing a fine or granting a prize money when there is no accident for a certain period of time.

Exemplarily, the fact that the management device 341 manages the damage information history of the taxi vehicle for each driver in consideration of the driver information means that the first damage information obtained based on the first damage image calculated before driving the first taxi driver and managing the second damage information obtained based on the second damage image calculated after driving of the first taxi driver in consideration of time.

As another example, the fact that the management device 341 manages the damage information history of the taxi vehicle in consideration of the driver information means that the first taxi driver obtained based on the first damage image calculated before driving the first taxi. The first damage history of the first user obtained by comparing the damage information and the second damage information obtained based on the second damage image calculated after the first taxi driver drove the first taxi, and the first taxi driver Third damage information obtained based on the third damage image calculated before driving the second taxi and fourth damage information obtained based on the fourth damage image calculated after the first taxi driver drove the second taxi It may include managing the second damage history of the first user obtained by the comparison in consideration of time.

3.2.3.3 Damage history management Example 2 - Management by vehicle

Damage information history can be managed for each vehicle.

According to an embodiment, the management device 341 may manage damage information records of taxi vehicles for each vehicle. The management device 341 manages the history of damage information of taxi vehicles for each vehicle, thereby providing an effect of easily grasping damage information generated from the past to the present in the taxi vehicle in a time-sequential manner. In addition, the management device 341 manages the damage information history of the taxi vehicle for each vehicle, so that the repair and inspection schedule of the taxi vehicle can be planned as a customized schedule for each vehicle according to the condition of the vehicle, so that efficient vehicle management can be achieved effect can be provided.

Exemplarily, the fact that the management device 341 manages the damage information history of the taxi vehicle for each vehicle means that the first damage information acquired based on the first image taken before driving the first taxi and after driving the first taxi It may include managing the second damage information obtained based on the captured second image in consideration of time.

As another example, the fact that the management device manages the damage information history of the taxi vehicle for each vehicle means that the first damage information and the first damage information obtained based on the first damage image calculated before the first taxi is operated by the first taxi driver The first damage history of the first taxi obtained by comparing the second damage information obtained based on the second damage image calculated after the taxi was operated by the first taxi driver, and the first damage history of the first taxi caused by the second taxi driver Comparing the third damage information obtained based on the third damage image calculated before driving and the fourth damage information obtained based on the fourth damage image calculated after the first taxi is operated by the second taxi driver It may include managing the obtained second damage history of the first taxi in consideration of time.

According to another embodiment, the management device 341 may manage damage information records of parked vehicles for each vehicle.

Illustratively, the management device 341 managing the damage information history of the parked vehicle for each vehicle means that the first damage information and the first parking vehicle obtained based on the first damage image calculated when the first parked vehicle enters the parking lot It may include managing the second damage information obtained based on the second damage image calculated when the vehicle is taken out in consideration of time.

3.2.3.4 Vehicle information management embodiment

According to an embodiment, a vehicle damage management apparatus may be provided that acquires information related to damage outside a vehicle based on a vehicle image.

The vehicle damage management apparatus includes an image acquisition unit configured to acquire at least one target vehicle image obtained by photographing the exterior of the target vehicle, and damage information related to exterior damage of the target vehicle based on the at least one target vehicle image. A processing unit may be included.

The at least one target vehicle image may include a first target vehicle image obtained by photographing one side of the target vehicle. One side of the target vehicle may be the front or rear of the vehicle. The at least one target vehicle image may include a vehicle image representing at least one of the front, rear, left side, right side, left front side, left rear side, right front side, right rear side, and top side of the vehicle.

The first target vehicle image may include a first area corresponding to the target vehicle and a second area where identification information of the target vehicle is located. The first area may be larger than the second area. The first region may include the second region.

The processor may obtain vehicle type information of the target vehicle determined based on the second area and the damage information obtained based on the vehicle type information and the first target vehicle image and related to the first area. For example, the damage information may indicate whether there is damage to the vehicle included in the first target vehicle image, the location, degree, size, type, area, and the like of the damage.

The vehicle damage management apparatus may further include an output unit that outputs the damage information. The output unit may output the damage information of the target vehicle. The output unit may output damage information through means such as image, video, and audio. The output unit may output damage information to an external device through a communication means.

The damage information may include damage area information corresponding to the damage of the target vehicle. The output unit may output the target vehicle image and the damaged area information overlapping the target vehicle image.

The vehicle damage management apparatus may further include an input unit that obtains a user input. The processing unit may obtain a user response to the damage information through the input unit in response to the output of the damage information.

The processing unit may obtain identification information of the target vehicle determined based on the second region. The processing unit may obtain the license plate number or VIN of the target vehicle, which is determined based on the region corresponding to the license plate or VIN of the vehicle.

The processing unit obtains identification information of the target vehicle determined based on the second area, and obtains a damage history of the target vehicle corresponding to the identification information of the target vehicle;

The damage history may be obtained based on a second target vehicle image taken prior to the first target vehicle image and may include existing damage information related to the exterior damage of the target vehicle. Existing damage information is damage information obtained based on a second target vehicle image taken before the first target vehicle image is captured, and may mean damage information other than new damage information to be determined through the first target vehicle image. can

The processing unit may obtain new damage information corresponding to a point in time when the first target vehicle image of the target vehicle was captured, based on the existing damage information and the damage information.

The damage information may include first damage information related to the second area included in the first target vehicle image and second damage information previously stored in association with identification information of the target vehicle obtained based on the first area. can be created based on

The at least one target vehicle image may further include a second target vehicle image obtained by photographing the other side of the target vehicle. Here, the second target vehicle image may be an image obtained by photographing an area not including identification information of the target vehicle. For example, if the first target vehicle image is a front or rear image obtained by photographing an area including a license plate of the vehicle, the second target vehicle image may be an image obtained by photographing the side of the vehicle.

The second target vehicle image may include a third area corresponding to the target vehicle. The processing unit obtains vehicle model information of the target vehicle determined based on the second region of the first target vehicle image and the damage information obtained based on the vehicle model information and the second target vehicle image and associated with the third region can do.

The processing unit may, based on the first target area image and the vehicle type information, a first part area included in the first area and corresponding to the first part of the target vehicle and a first part area included in the first area and corresponding to the first part of the target vehicle. Part information including a second part region corresponding to the second part may be obtained.

The processing unit may obtain part area information corresponding to the damage based on the damage information and the part information.

The processing unit may obtain the second area corresponding to the license plate of the target vehicle from the first target vehicle image, and obtain the vehicle type information based on the second area corresponding to the license plate.

The processing unit may obtain the damage information based on the target vehicle image by using a neural network model learned to obtain the damage information based on the target vehicle image. The neural network model may be trained to acquire damage information based on a vehicle image using learning data including a damaged area or a vehicle image labeled with damage information. The damage information may include area information indicating an area of the first area that corresponds to damage outside the vehicle.

According to another embodiment, a vehicle damage management method using a device including an image acquisition unit acquiring a vehicle image and a processing unit obtaining damage information may be provided.

31 is a diagram for explaining a vehicle damage management method according to an embodiment. Referring to FIG. 31 , the vehicle damage management method includes acquiring a target vehicle image (S3110), obtaining vehicle model information of the target vehicle (S3130), and acquiring damage information based on the target vehicle image and vehicle model information. (S3150) may be included.

The vehicle damage management method may include acquiring at least one target vehicle image obtained by photographing the exterior of the target vehicle through an image acquisition unit ( S3110 ). The at least one target vehicle image includes a first target vehicle image obtained by photographing a side of the target vehicle, and the first target vehicle image includes a first region corresponding to the target vehicle and identification information of the target vehicle. may include a second area where is located.

The vehicle damage management method may include obtaining vehicle type information of the target vehicle determined based on the second area through a processing unit ( S3130 ).

Acquiring damage information related to exterior damage of the target vehicle based on vehicle model information and the first target vehicle image through a processing unit ( S3150 ).

The vehicle damage management method includes outputting damage information; may further include. The damage information may include damage area information corresponding to the damage of the target vehicle, and outputting the damage information may further include outputting the target vehicle image and the damage area information overlapping the target vehicle image.

According to an embodiment, a vehicle damage management method may be provided in which identification information is obtained and based on the identification information, damage history and/or new damage information of the vehicle is acquired. 32 is a diagram for explaining a vehicle damage management method according to an embodiment. Referring to FIG. 32 , the vehicle damage management method includes obtaining identification information of a target vehicle (S3171), acquiring a damage history of the target vehicle (S3173), and/or acquiring new damage information (S3175). may further include.

The vehicle damage management method may further include obtaining identification information of the target vehicle determined based on the second region through a processing unit ( S3171 ).

The vehicle damage management method may further include acquiring a damage history of the target vehicle corresponding to identification information of the target vehicle through the processing unit ( S3173 ). The damage history may be obtained based on a second target vehicle image taken prior to the first target vehicle image and may include existing damage information related to the exterior damage of the target vehicle.

The vehicle damage management method includes obtaining new damage information corresponding to a time point when the first target vehicle image of the target vehicle was captured based on the existing damage information and the damage information through the processing unit (S3175) may further include.

The damage information may include damage area information included in the first area and corresponding to the damage of the target vehicle.

The at least one target vehicle image may further include a second target vehicle image obtained by photographing another side of the target vehicle different from the one side, and the second target vehicle image may include a third area corresponding to the target vehicle. can

The obtaining of the damage information may include obtaining the damage information based on vehicle type information of the target vehicle determined based on the second area of the first target vehicle image and the vehicle type information and the second target vehicle image, and the third area It may further include obtaining the damage information related to.

According to an embodiment, a vehicle damage management method may be provided in which part information is obtained and part area information corresponding to damage is obtained based on the part information. The vehicle damage management method may further include obtaining part information of the target vehicle (S3191) and obtaining part area information corresponding to the damage (S3193).

The vehicle damage management method includes a first part region included in the first region and corresponding to a first part of the target vehicle and the first region based on the first target region image and the vehicle type information through a processing unit. The method may further include obtaining part information including a second part area included in and corresponding to the second part of the target vehicle (S3191).

The vehicle damage management method may further include obtaining part area information corresponding to the damage based on the damage information and the parts information through the processing unit ( S3193 ). Acquiring part area information corresponding to the damage may include acquiring the name, identification code, number, and the like of the part where the damaged area is located. Acquiring part area information corresponding to the damage may include acquiring part-damage information described herein.

The vehicle damage management method may further include acquiring the second area corresponding to the license plate of the target vehicle from the first target vehicle image through a processing unit. Obtaining vehicle model information may further include obtaining the vehicle model information based on the second region corresponding to the license plate.

The obtaining of the damage information may further include obtaining the damage information based on the target vehicle image by using a neural network model learned to acquire the damage information based on the target vehicle image.

The vehicle damage management method may further include an input unit that obtains a user input. In this case, the vehicle damage management method may further include obtaining a user response to the damage information through the input unit in response to the processing unit outputting the damage information.

The above-described vehicle damage management method may be provided as a computer readable recording medium storing a program for performing the method.

3.2.4 Vehicle information output

3.2.4.1 Output information

The on-site vehicle management system may output vehicle information and auxiliary information obtained based on the vehicle information through the display of the management device 341 . In addition, the management device 341 may output vehicle information and auxiliary information through an electronic device connected to the management device 341 .

The management device 341 may output vehicle information such as parts information, damage information, vehicle type information, and identification information. The vehicle management system may output user information and/or location information together.

The management device 341 may output a vehicle image and vehicle information corresponding to the vehicle image. The vehicle management system may output a plurality of vehicle images and vehicle information corresponding to each vehicle image. The plurality of vehicle images may be images taken from various angles with respect to the vehicle.

Hereinafter, vehicle information output screens of the parking management system and the taxi vehicle management system will be described with reference to FIGS. 44 to 47 .

3.2.4.2 Vehicle information output Embodiment 1 - Parking control company

44 to 48 are views for explaining screens for outputting vehicle information and/or auxiliary information in the management device 341 according to an embodiment.

According to an embodiment, the management device 341 may output acquired vehicle information and/or auxiliary information acquired based on the vehicle information. The management device 341 may output vehicle information and/or auxiliary information to a manager's information communication device. Although not shown in the drawing, the above-described vehicle information and/or auxiliary information may be output to an information communication device of a vehicle driver.

The management device 341 may output vehicle information and/or auxiliary information when new damage information is detected on a parked vehicle. In addition, the management device 341 may output vehicle information and/or auxiliary information based on an input through the user interface of the management device 341 when the driver claims that the parked vehicle has been damaged while parking. . In addition, the management device 341 may output vehicle information and/or auxiliary information for managing a parked vehicle even when no new damage information is detected or the driver does not claim damage occurred during parking.

In addition, the management device 341 may output that additional damage has occurred to the vehicle driver in a state where the circuit breaker PB is not opened when new damage information of the vehicle being pulled out is detected. In this case, the management device 341 may output that additional damage has occurred to the vehicle driver's information and communication device, and when the vehicle driver proceeds to confirm the additional damage, the management device 341 operates a circuit breaker (PB) may be opened.

Alternatively, the management device 341 may deliver to the vehicle driver's information communication device that additional damage has occurred after opening the circuit breaker (PB) when new damage information of the vehicle being pulled out is detected.

44 is a diagram for explaining a screen for outputting damage information among output screens of the management device 341.

The output screen according to an embodiment includes a vehicle image display unit (CI), a direction indicator (DI) indicating a capturing direction of the vehicle image, a non-image information display unit (IF) related to the vehicle, and a damage information table (which represents damage information of the vehicle). DT) may be included.

The vehicle image display unit CI according to an exemplary embodiment may display an image from which noise is removed. The vehicle image display unit CI may display a vehicle image captured when leaving the parking area. The vehicle image display unit CI may display a vehicle image including information about the damaged area of the vehicle captured when leaving the parking area. The damaged area information may include an existing damaged area ED and a new damaged area ND. The existing damaged area ED and the new damaged area ND may be displayed differently. Also, the existing damaged area ED and the new damaged area ND may be displayed as bounding boxes of different colors or shapes.

In addition, the vehicle image displayed on the vehicle image display unit CI may be output in a form that can be enlarged or reduced by the user.

According to an exemplary embodiment, the direction indicator DI may indicate a photographing direction of a displayed vehicle image. In addition, the direction indicator DI may indicate the left direction of the vehicle, and the vehicle image display unit CI may display a vehicle image captured from the left side of the vehicle. Vehicle images photographed in different directions may correspond to each of a plurality of directions indicated by the direction indicator DI. A corresponding vehicle image may be displayed in response to a user input for selecting a direction of the direction indicator DI.

The non-image information display unit IF according to an embodiment may display non-image information related to a vehicle. Referring to FIG. 44 , the non-image information display unit IF may display a time point at which a vehicle image was captured, a vehicle number, a vehicle model, and the like. Illustratively, the timing at which the vehicle image is captured may mean a time of entering the parking area and a time of leaving the parking area. In addition, the non-image information display unit IF may display vehicle number and vehicle model name.

The damage information table DT according to an embodiment may indicate the number of damages for each part corresponding to the vehicle image. The part corresponding to the image may refer to a part corresponding to the direction of the currently displayed vehicle image. For example, since an image of the left side of the vehicle is displayed in FIG. 44 , the damage information table DT may indicate the number of damages of parts located on the left side of the vehicle. The number of damages per part may include the number of existing damages and the number of new damages (separated by +) per part. Referring to FIG. 44 , the damage information table DT may indicate that existing damage exists on the left front door, left door handle, and left rear fender, and new damage exists on the left front door and left rear fender.

Meanwhile, the damage information table DT is only one embodiment of the damage information display unit, and information on existing damage or new damage may be displayed in other forms.

45 is a view for explaining a screen for selecting whether or not a user or manager consents to damage information among output screens of the management device 341, and FIG. 46 is a diagram for damage information among output screens of the management device 341. It is a drawing for explaining a screen including detailed contents.

Referring to (a) of FIG. 45 , a screen for outputting damage information among output screens of the management device 341 according to an embodiment may include a detailed check button CO. If the user or manager does not agree with the information on the new damaged area (ND) or wants to obtain more detailed information even if they agree, they can click the detailed confirmation button (CO) included in the vehicle image display unit (CI).

When the user or manager clicks the detailed confirmation button CO, the output screen of the management device 341 may display a damage image calculated upon entering and a damage image calculated upon leaving the vehicle. In this case, the damage image calculated when entering the vehicle and the damage image calculated when exiting the vehicle may be vehicle images photographed from directions corresponding to each other. As another example, the damage image calculated upon entry and the damage image calculated upon exit include at least one main part of the vehicle, and at least one main part of the vehicle may be common.

Alternatively, the output screen of the management device 341 may display an image in which incident value damage information is mapped to effective image data derived upon entering a vehicle. In addition, the output screen of the management device 341 may display an image in which damage information upon exiting the vehicle is mapped to effective image data derived upon exiting the vehicle.

An existing damaged area ED may be included in the damaged image calculated upon vehicle entry, and a new damaged area ND may be included in the damaged image calculated upon vehicle exit. That is, when the user or manager clicks the detailed check button, the user or manager can obtain more specific information about the new damage by comparing the vehicle image captured when entering the vehicle with the damage image calculated when leaving the vehicle.

Referring to (b) of FIG. 45 , a screen for outputting damage information among output screens of the management device 341 according to another embodiment may include a selection window (CO) for selecting consent to new damage information. can The user or administrator can click the 'Confirm' button if they agree with the information on the new damaged area (ND), and can click the 'Objection' button if they do not agree.

Referring to FIG. 46 , when the user or manager clicks the 'Objection' button, the output screen of the management device 341 may display a damage image calculated upon entering and a damage image calculated upon exiting.

47 is a view for explaining an output screen including information about a damage history among output screens of the management device 341.

Referring to FIG. 47 , an output screen including information about damage history according to an embodiment may include a vehicle image display unit CI, a vehicle-related non-image information display unit IF, and a damage history display unit DH. can

Since the vehicle image display unit CI and the non-image information display unit IF have been described above, a detailed description thereof will be omitted.

The damage history display unit DH may display damage information occurring to the vehicle from the past to the present. In addition, the damage history display unit DH may display information including a date of occurrence of damage, a time of occurrence of damage, the number of damages, a location of damage, and the like regarding damages occurring to the corresponding vehicle from the past to the present.

3.2.4.3 Example 2 of Vehicle Information Output - Taxi Company

The information output on the display of the management device 341 of the taxi vehicle management system corresponds to the information output from the parking management system described in 3.2.4.2. Therefore, descriptions overlapping with those described in the parking management system will be omitted.

According to an embodiment, the management device 341 of the taxi vehicle management system may output acquired vehicle information and/or auxiliary information obtained based on the vehicle information. The management device 341 may output vehicle information and/or auxiliary information to a manager's information communication device. Although not shown in the drawing, the above-described vehicle information and/or auxiliary information may be output to a taxi driver's information communication device.

48 is a diagram for explaining an output screen including information about a damage history among output screens of the management device 341.

Referring to FIG. 48, the screen for classifying and outputting the history of damage to the taxi by driver may include a non-image information display unit (IF) related to the driver, a damage history display unit (DH), and a vehicle image display unit (CI). .

The non-image information display unit IF according to an embodiment may display information about the driver, for example, the driver's name, driver's license information, driver's address, and the like. In addition, the non-image information display unit IF may display information about the total mileage, total driving time, total number of accidents, total number of repairs/replacements, total repair/replacement costs, and the like.

The damage history display unit DH according to an embodiment may display damage information caused by a corresponding driver from the past to the present. In addition, the damage history display unit DH may display information about damage caused by the driver from the past to the present, such as a vehicle number where the damage occurred, a date the damage occurred, the number of damages, and the location of the damage.

According to an embodiment, the vehicle image display unit CI may display detailed information about damage information selected by the driver or manager when the driver or manager selects one of the corresponding driver's damage histories. In this case, the details of the damage information selected by the driver or manager may include information about the insurance number of the vehicle in which the accident occurred, whether to replace/repair parts, the price of parts replacement/repair, and the like.

In addition, the damage history information output screen may classify and output the history of damage occurring to the taxi for each vehicle. The management device 341 may link and display the driver when damage occurs for each damaged area in the taxi.

4. Mobile base vehicle management system

4.1 Mobile system

According to one embodiment, a vehicle management system for managing a state of a vehicle using an image captured by a mobile device may be provided.

49 is a diagram for explaining the configuration of a mobile-based vehicle management system.

Referring to FIG. 49 , a mobile-based vehicle management system according to an embodiment may include a terminal 490 and a server 499.

The terminal 490 may be connected to the server 499. When the terminal 490 is connected to the server 499, the terminal 490 can exchange data with the server 499. In this case, the server 499 may be connected to the plurality of terminals 490 and receive and process data from the plurality of terminals 490 .

The terminal 490 may include at least one camera module 491 , a controller 492 , a storage unit 493 , a communication unit 494 , an input unit 495 , and an output unit 496 .

The camera module 491 may capture the exterior of the vehicle. The camera module 491 may capture the exterior of the vehicle before and after using the shared vehicle.

The storage unit 493 may store photographing data captured by the camera module 491 . Also, the storage unit 493 may store non-image data.

The communication unit 494 is a component that transmits/receives data. The terminal 490 and the server 499 may be connected to each other through the communication unit 494 to exchange data.

The input unit 495 is a component that receives data from a user or administrator. The data may include vehicle-related information, user information, and the like.

The output unit 496 is a component that outputs vehicle-related information to a user or manager, and may refer to a module such as a display.

The control unit 492 may control the storage unit 493, the communication unit 494, the input unit 495, and the output unit 496. The control unit 492 controls the shooting data captured by the camera module 491 to be stored in the storage unit 493, and controls data to be transmitted and received in the communication unit 494 in the form of various embodiments, and the input unit 495 ) to receive information about the vehicle and user information, and control to output information about the vehicle to the user or manager through the output unit 496.

4.2 Mobile-based fleet management process

According to an embodiment, a vehicle management process for managing a state of a vehicle using an image captured by a mobile device may be provided. The mobile-based vehicle management process may be performed by the aforementioned mobile-based vehicle management system.

50 is a diagram for explaining a series of steps performed by a mobile-based vehicle management process.

Referring to FIG. 50 , the mobile-based vehicle management process includes acquiring vehicle data captured in a mobile device (S510), acquiring vehicle information based on the obtained vehicle data (S530), and managing vehicle information (S530). S550) and outputting vehicle information (S570). Hereinafter, each step will be described in more detail.

4.2.1 Mobile Vehicle Data Acquisition

The mobile-based vehicle management process may include obtaining vehicle data necessary for vehicle management (S510). Vehicle data may be prepared in various forms. For example, vehicle data may be image data or non-image data. Vehicle data may be pre-stored in the device or obtained from the outside.

The acquisition of vehicle data and vehicle data will be described in more detail below with specific examples.

4.2.1.1 Image Data Acquisition

Obtaining image data according to an embodiment may include acquiring image data including a plurality of images and extracting valid image data from among the plurality of acquired images. The video data and effective image data are components corresponding to the video data and effective image data described in 3.2.1.1, respectively, and since the contents have been described above, overlapping contents will be omitted.

51 is a diagram for explaining a recapturing request when vehicle damage information included in image data is not sufficient, according to an exemplary embodiment.

The mobile-based vehicle management process can operate normally only when the image data includes information on the number of damages, types of damages, and degrees of damages, locations where damages occur, and information on parts where damages occur. Accordingly, when the information on the vehicle damage is not sufficiently expressed in the image data, the terminal 490 may request the user or manager to re-photograph.

That is, image data photographed by the terminal 490 is transmitted to the server 499, and the server 499 determines whether or not information on vehicle damage is sufficiently expressed in the received image data, and the result is referred to as the above. It can be transmitted to the terminal 490. The terminal 490 may display the determination result to the user through the output unit 496 . The terminal 490 may notify the user that the captured image data is appropriate when information on vehicle damage is sufficiently expressed, and when information on vehicle damage is not sufficiently expressed in the captured image data. Users may be notified that re-shooting is necessary.

Alternatively, the terminal 490 may determine whether information on vehicle damage is sufficiently expressed in the captured image data.

Referring to (a) of FIG. 51 , the image data MBB1 captured by the camera module 491 does not clearly include information about the number of damages, types of damages, and degrees of damages. In addition, referring to (b) of FIG. 51 , as a result of zooming in on the damaged area, the image data (MBB2) photographed by the camera module 491 includes information about the location of the damage in the exterior of the vehicle, information on the damaged part, and the like. is not explicitly included.

In this case, the control unit 492 of the terminal 490 informs the user or manager through the output unit 496 of the number of damages, types of damages, and degree of damages to the image data MBB3 as shown in (c) of FIG. 51 . A message requesting re-photographing can be displayed so that information about the image, the location where the damage occurred, and the information on the part where the damage occurred are all included.

Alternatively, the control unit 492 of the terminal 490 may display a message requesting re-photographing of the existing damaged portion if the photographing of the existing damaged portion stored in the storage unit 493 does not proceed.

Meanwhile, image data may refer to data captured by the camera module 491 between a plurality of specific points of time. In this case, the plurality of specific viewpoints may mean different viewpoints. Hereinafter, the plurality of specific viewpoints will be described in more detail with reference to the drawings by way of specific examples.

52 is a diagram for explaining that image data is captured by the camera module 491 at a specific point in time according to an exemplary embodiment.

Image data may include a plurality of images. Image data may include a first image and a second image. The first image is an image captured by the terminal 490 located at a first point having a first distance D1 from a specific point of the vehicle being photographed. The second image is an image captured by the terminal 490 located at a second point having a second distance D2 from the specific point of the vehicle. In this case, the first distance D1 and the second distance D2 may be different.

For example, the image data may be an image captured by a user's manipulation. The first image and the second image may be images captured by a user's manipulation. That is, a first image may be captured by a user's first manipulation of the terminal 490, and a second image may be captured by a second manipulation.

Alternatively, the terminal 490 may extract a part of image data captured by a user's manipulation. That is, the terminal 490 is photographed in the form of a video by a user's manipulation, and the terminal 490 may extract video data of a main part of the video data as a first image and a second image. Alternatively, the server 499 may receive image data, and the server 499 may extract image data obtained by photographing a main part of the image data as a first image and a second image.

When image data is acquired by the camera module 491 at a point having a first distance D1 and a second distance D2 different from the first distance D1, the image data includes information on vehicle damage as well as corresponding damage. Various types of information other than information may be included. Accordingly, the terminal 490 can detect valid image data from which noise is removed from the image data, that is, more accurate information about damage to the vehicle is included.

In addition, the first image is an image in which a first area of the exterior of the vehicle is captured, and at least one main part of the vehicle is included in the first area. The second image is an image in which a second area of the exterior of the vehicle is captured, and at least one main part of the vehicle is included in the second area. In this case, at least some of the main parts of the vehicle photographed by the terminal 490 may be commonly included in the first area and the second area.

53 is a view for explaining a shooting guide provided based on target vehicle information when a vehicle is photographed with the camera module 491.

Referring to (a) of FIG. 53 , the terminal 490 may display a shooting guide through the output unit 496 based on target vehicle information when a user or manager photographs a vehicle with a camera module 491. there is. The information of the target vehicle may be obtained using a neural network model learned to acquire information input by a vehicle user, reservation information of a vehicle user, or vehicle information. For example, when a vehicle user makes a reservation for a specific vehicle type before using the vehicle, the terminal 490 may provide the user or manager with a shooting guide for the corresponding vehicle model through the output unit 496 based on the user's reservation information. there is. As another example, when a vehicle user photographs a vehicle with the camera module 491 for vehicle use, the terminal 490 acquires information about the vehicle model through a neural network model trained to obtain vehicle information, and then obtains information about the vehicle model. A shooting guide may be provided to a user or manager through the output unit 496 . Meanwhile, the photographing guide shown in (a) of FIG. 53 is exemplary, and the photographing guide according to an embodiment may include various types of known photographing guides.

Referring to (b) and (c) of FIG. 53 , when a user photographs a vehicle based on the above-described photographing guide, when the vehicle is properly positioned within a predetermined ratio or more, the control unit 492 of the terminal 490 ) may enable the photographing button to enable the user to photograph and store the vehicle.

In addition, when the user photographs a vehicle based on the above-described photographing guide, when the vehicle is properly positioned within a predetermined ratio or more in the photographing guide, the controller 492 of the terminal 490 automatically sets the camera module 491 to the vehicle. After the external appearance is controlled to be photographed, the photographed image may be controlled to be stored in the storage unit 493 of the terminal 490 .

Meanwhile, although not shown in FIG. 53 , the terminal 490 may provide not only a photographing guide for photographing the entire vehicle, but also a photographing guide for each part for photographing a specific part of the vehicle. The photographing guide for each part may be obtained by using a neural network model learned to acquire information input by a vehicle user, reservation information of a vehicle user, or vehicle information, and since this has been described above, redundant information will be omitted.

54 is a diagram for explaining a photographing map provided when a vehicle is photographed with the camera module 491.

Referring to FIG. 54 , the terminal 490 may display a shooting map to the user through the output unit 496 . When a user photographs a vehicle with the camera module 491, the terminal 490 may display a photographing map to the user through the output unit 496, which guides the photographing of all major parts of the exterior of the vehicle. More specifically, the photographic map may include a UI for inducing photographing of the vehicle in various directions so that all major parts of the exterior of the vehicle may be photographed. Illustratively, the main parts of the exterior of the vehicle may include the front and back of the vehicle, the front side of the passenger seat, the back side of the passenger seat, the front side of the driver's seat, and the back side of the driver's seat.

The photographing map may include a guide unit GU and a photographing unit CB. Hereinafter, specific embodiments of the guide unit GU and the photographing unit CB will be described with reference to the drawings.

When a user photographs the exterior of a vehicle with the camera module 491, the terminal 490 may guide the photographing through the guide unit GU so that main parts of the exterior of the vehicle may be photographed. The guide unit GU may include at least one guide so that all major parts of the exterior of the vehicle can be photographed. The terminal 490 may induce vehicle photographing through the guide unit GU so that the vehicle may be photographed in different directions.

Exemplarily, the guide unit GU includes a first guide for inducing a photograph of the front of the vehicle, a second guide for inducing a photograph of the front side of the passenger seat, a third guide for inducing a photograph of the back of the passenger seat, and a photograph of the rear of the vehicle. A fourth guide for guiding, a fifth guide for guiding shooting of the back of the driver's seat, and a sixth guide for guiding shooting of the front of the driver's seat may be included.

Meanwhile, the photographing unit CB may include a function of photographing the exterior of the vehicle and storing the photographed image. The photographing unit CB may include a photographing button capable of photographing a main part of the exterior of the vehicle. More specifically, the photographing unit CB may include a photographing button capable of photographing the exterior of a vehicle guided by the guide when the user selects one of the guides described above.

For example, when the user selects the first guide for inducing photographing of the front of the vehicle in the guide unit GU, the terminal 490 displays a photographing button capable of photographing the front of the vehicle through the output unit 496. This activated photographing unit CB can be displayed. At this time, when the user clicks the photographing button to photograph the front of the vehicle, the controller 492 of the terminal 490 may classify the captured image as the front of the vehicle and store the captured image in the storage unit 493.

Meanwhile, image data obtained by the camera module 491 may include a plurality of images. At this time, valid image data may be extracted from among the plurality of images in order to allow the mobile-based vehicle management process to be normally performed.

Since the step of extracting valid image data suitable for the mobile-based vehicle management process from the image data corresponds to step S3513 of FIG. 36 , overlapping details will be omitted.

4.2.1.2 Non-Image Data Acquisition

According to one embodiment, vehicle data in the form of non-images other than images may be obtained. Vehicle data may be non-image data. Vehicle data may be non-image data including vehicle-related information. Since the non-image data has been described above in 3.2.1.2, overlapping content will be omitted.

The non-image data may be stored or acquired in advance. The non-image data may be acquired by the camera module 491 . The non-image data may be obtained based on photographing data captured by the camera module 491 . The management device 341 may obtain non-image data based on the image data.

4.2.1.3 Data Processing

In the mobile-based vehicle management process, processing of the acquired data may be performed.

In the mobile-based vehicle management process, pre-processing (or post-processing) may be performed on the acquired data. The terminal 490 may perform pre-processing on at least one effective image data among image data. The terminal 490 may perform preprocessing based on the first image and the second image among the video data.

The terminal 490 may perform noise removal processing based on a plurality of images captured between a plurality of specific points. The terminal 490 may perform noise removal processing based on a plurality of valid image data.

An image on which the above-described noise removal process is performed may be defined as a damaged image. The terminal 490 may determine vehicle information including damage to the vehicle based on the damage image.

When the camera module 491 is installed in a noise-free environment, the management device 341 may omit the noise removal process. In this case, acquisition of vehicle information to be described later may be performed based on valid image data. The damaged image at this time may be the valid image data.

The terminal 490 may perform noise removal processing for removing reflected light or the like based on a plurality of valid image data. The terminal 490 obtains a plurality of valid image data, and an object whose position is changed (eg, a reflective area corresponding to reflected light) and an object whose position is not changed (eg, a vehicle area or a damaged area corresponding to damage to the vehicle). ), it is possible to remove an object whose position is changed or an object whose position is not changed. For example, the terminal 490 obtains a vehicle image from which the reflective region is removed based on effective image data including a damaged region corresponding to a damaged part of the vehicle and a reflective region corresponding to light reflected by the surface of the vehicle. processing can be performed. The effective image data may be image data including a main region.

Since the terminal 490 is not captured in a fixed form, a first image having a first distance from the main part and a second image having a second distance to the main part are between the camera module 491 and the main part. It is highly likely that the distance of That is, the first distance and the second distance may be different. Accordingly, when the first image and the second image are captured by the first and second manipulations while the user is carrying the terminal 490, and when the user is carrying the terminal 490, the first image and the second image are captured. When a video is taken by manipulation and the first image and the second image are extracted based thereon, there is a high possibility that the first distance and the second distance are different.

The first distance may be a distance between the camera module 491 and the first feature point of the vehicle when a first image is captured, and the second distance may be a distance between the camera and the first feature point when a second image is captured. may be the distance between

Accordingly, effective image data capable of performing noise processing can be obtained without extra effort, and the terminal 490 can perform noise removal processing based on this.

Although the noise removal processing is described as being performed in the terminal 490, the noise removal processing may be performed in the server 499.

4.2.2 Obtaining mobile vehicle information

The mobile-based vehicle management process may include obtaining vehicle information based on vehicle data.

The vehicle information may include parts information related to parts constituting the target vehicle, damage information related to internal/external damage of the target vehicle, vehicle model information related to the type of the target vehicle, and/or identification information for identifying the target vehicle. .

Vehicle information may be extracted from vehicle data or obtained by processing vehicle data. Vehicle information may be obtained using an algorithm designed to obtain vehicle information or a neural network model trained to obtain vehicle information, with vehicle data as input data.

Acquisition of vehicle information based on vehicle data has been described above in 2.3, so duplicated content is omitted.

According to an embodiment, the terminal 490 may obtain vehicle information based on the damaged image. The terminal 490 may obtain damage information based on the damage image.

According to an embodiment, in the case of a carsharing company, the terminal 490 may obtain shared vehicle information using a pre-stored database. The shared vehicle information may include vehicle type information indicating a model, manufacturer, model name, and year of the shared vehicle. In addition, the shared vehicle information may further include parts information indicating the number, type, name, and identification information of parts constituting the shared vehicle. In addition, the shared vehicle information may further include additional information including personal information of the shared vehicle user or an accident history of the shared vehicle.

The shared vehicle information may include additional damage information generated inside/outside the target vehicle. The additional damage information may be calculated based on a damaged image of the vehicle before the user uses the vehicle and a damaged image of the vehicle after the user uses the vehicle. The additional damage information may be calculated by comparing a damaged image of the vehicle before the user uses the vehicle and a damaged image of the vehicle after the user uses the vehicle.

The additional damage information may be calculated based on damage information calculated based on a damaged image of the vehicle before the user uses the vehicle and damage information calculated based on the damaged image of the vehicle after the user uses the vehicle.

The terminal 490 stores at least one of a damage image and damage information of the vehicle before the user uses the vehicle, and retrieves it after the user uses the vehicle to retrieve the image and damage information of the vehicle after the user uses the vehicle. Additional damage information may be calculated by comparing with at least one of the above.

The terminal 490 may store non-image data of the vehicle before the user uses the vehicle, and the damaged image and damage information in conjunction with each other. For example, the terminal 490 may store a damage image and damage information of a vehicle before the user uses the vehicle in association with the license plate number of the vehicle.

When the user has finished using the vehicle, the terminal 490 loads a damage image and damage information linked to the license plate number of the vehicle based on the recognized license plate number, and loads the damaged image and damage information of the vehicle after the user uses the vehicle. Additional damage information may be calculated by comparing the damage image and the damage information.

4.2.3 Vehicle information management

The mobile-based vehicle management process may include managing vehicle information. Managing vehicle information may include managing the aforementioned parts information, damage information, vehicle type information, and/or identification information.

Also, managing the vehicle information may include managing the vehicle information time-sequentially. Managing vehicle information may include classifying vehicle information. Managing vehicle information may include managing one or more pieces of information in association with each other. The vehicle information management may be performed by the server 499 .

Hereinafter, management of vehicle information will be described with reference to some embodiments.

4.2.3.1 History management

Managing vehicle information may include history management of vehicle information. History management of vehicle information may mean time-sequential listing and/or management of one or more pieces of vehicle information obtained at different times.

According to an embodiment, managing vehicle information in a mobile-based vehicle management process may mean managing damage information representing damage to a specific vehicle together with time information representing a time when the damage information was obtained. there is. For example, managing vehicle information acquires first information indicating that a first damage has occurred to a first vehicle at a first time point, and second damage has occurred to the first vehicle at a second time point after the first time point. It may include obtaining second information indicating, and managing the first information and the second information in chronological order.

Exemplarily, managing vehicle information in a carsharing process may mean managing damage information representing damage to a shared vehicle together with time information representing a time when the damage information was acquired.

More specifically, managing vehicle information in the carsharing process means obtaining first information indicating that a first damage has occurred to the shared vehicle before the first user uses the first vehicle, and the first user uses the first vehicle. obtaining second information indicating that a second damage has occurred to the shared vehicle after the first user obtains third information indicating that a third damage has occurred to the shared vehicle before the first user uses the second vehicle; After obtaining fourth information indicating that fourth damage has occurred in the shared vehicle after using the second vehicle, managing the first information, second information, third information, and/or fourth information in chronological order. can

In addition, managing vehicle information in a carsharing process means obtaining first information indicating that a first damage has occurred to a shared vehicle after a first user uses a shared vehicle, and obtaining information on a shared vehicle before a second user uses a shared vehicle. After obtaining second information indicating that a second damage has occurred and obtaining third information indicating that a third damage has occurred in a shared vehicle after the second user uses the shared vehicle, the first information, the second information and / or managing the third information in chronological order. In this case, the first user may refer to a person who prefers to use the shared vehicle over the second user.

4.2.3.2 Damage history management Example 1 - By driver

In the mobile-based vehicle management process, the history of damage information can be managed. History management of damage information may include acquiring new damage information for newly occurring new damage. History management of damage information may include obtaining new damage information based on a plurality of images obtained by photographing the same vehicle at various times.

The damage information history may be managed for each user in consideration of user information.

According to an embodiment, the terminal 490 may manage damage information records of shared vehicles for each user in consideration of user information. The terminal 490 manages the damage information history of the shared vehicle for each user, so that when damage occurs to the shared vehicle, responsibility can be clarified, and which user has caused current or past damage to the shared vehicle. It is easy to grasp and can provide an effect of reducing the cost of managing a shared vehicle accordingly. In addition, the terminal 490 may provide an effect of performing an evaluation of the user by managing the history of damage information of the shared vehicle for each user.

Exemplarily, the terminal 490 managing the damage information history of the shared vehicle for each user in consideration of user information means that the first information indicating that the first damage occurred to the shared vehicle before the first user used the first vehicle managing additional damage information generated while the first user uses the first vehicle obtained by comparing the first user with second information indicating that second damage occurs in the shared vehicle after the first user uses the first vehicle; and Third information indicating that third damage occurred to the shared vehicle before using the second vehicle and fourth information indicating that fourth damage occurred to the shared vehicle after the first user used the second vehicle 1 This may mean managing additional damage information generated while using the second vehicle by the user.

4.2.3.3 Damage history management Example 2 - Management by vehicle

Damage information history can be managed for each vehicle.

According to an embodiment, the terminal 490 may manage damage information records of shared vehicles for each vehicle. Since the terminal 490 manages the damage information history of the shared vehicle for each vehicle, it is possible to provide an effect of easily grasping damage information generated in the shared vehicle from the past to the present in a time-series manner. In addition, the terminal 490 manages the damage information history of the shared vehicle for each vehicle, so that the repair and inspection schedule of the shared vehicle can be planned as a customized schedule for each vehicle according to the state of the vehicle, so that efficient vehicle management can be achieved effect can be provided.

Exemplarily, the terminal 490 managing the damage information history of the shared vehicle for each vehicle means that the first information indicating that the first damage occurs in the shared vehicle after the first user uses the shared vehicle is shared by the second user. This may refer to managing additional damage information generated during parking of the shared vehicle by comparing second information indicating that the second damage has occurred to the shared vehicle before using the vehicle.

As another example, the terminal 490 managing the damage information history of the shared vehicle for each vehicle means that the first information indicating that a first damage has occurred to the shared vehicle after the first user uses the shared vehicle or the shared vehicle of the second user The second information indicating that the second damage has occurred to the shared vehicle before use is compared with the third information indicating that the third damage has occurred to the shared vehicle after the second user has used the shared vehicle, and the second user is using the shared vehicle. It may mean managing additional damage information that has occurred.

4.2.4 Vehicle information output

4.2.4.1 Output information

The mobile-based vehicle management system may output vehicle information and auxiliary information acquired based on the vehicle information through the output unit 496 of the terminal 490 . In addition, the terminal 490 may output vehicle information and auxiliary information through a server 499 connected to the terminal 490 .

The terminal 490 may output vehicle information such as parts information, damage information, vehicle type information, and identification information. The vehicle management system may output user information and/or location information together.

The terminal 490 may output a vehicle image and vehicle information corresponding to the vehicle image. The vehicle management system may output a plurality of vehicle images and vehicle information corresponding to each vehicle image. The plurality of vehicle images may be images taken from various angles with respect to the vehicle.

Hereinafter, a vehicle information output screen of the shared vehicle management system will be described with reference to FIG. 55 .

4.2.4.2 Vehicle information output embodiment

55 is a diagram for explaining a screen for outputting vehicle information and/or auxiliary information in the terminal 490 according to an exemplary embodiment.

According to an embodiment, the terminal 490 may output acquired vehicle information and/or auxiliary information obtained based on the vehicle information. The terminal 490 may output vehicle information and/or auxiliary information to an information communication device of a user or manager.

When additional damage information is detected in the shared vehicle, the terminal 490 may output vehicle information and/or auxiliary information. In addition, the terminal 490 may output vehicle information and/or auxiliary information when additional damage is detected in the shared vehicle before/after the user uses the shared vehicle. In addition, the terminal 490 may output vehicle information and/or auxiliary information even when additional damage information is not detected for management of shared vehicles.

55(a) is a diagram for explaining a screen for outputting information on a current external state of a shared vehicle.

Referring to (a) of FIG. 55 , the shared vehicle exterior information screen according to an embodiment may include a damage information table and a damage image displaying damage information on the exterior of the vehicle.

The vehicle exterior information output screen may include a damage information table displaying information on damage occurring to the shared vehicle. The damage information table may include damaged vehicle parts, damage types, damage levels, and the number of damages.

The vehicle exterior information output screen may include a damage image on which damage information is displayed. The damaged image may include damaged area information, and the damaged area information may include an existing damaged area ED.

The vehicle appearance information output screen may include a selection window for selecting whether to consent to existing damage information. The user can click the 'Confirm' button if he or she agrees with the information about the damaged area, and can click the 'Objection' button if he or she does not agree. When the 'Confirm' button is clicked by the user, the terminal 490 may inform the server 499 that the collective agreement on the existing damage information has been performed.

Alternatively, the vehicle appearance information output screen may display a damage image in which damage information is displayed. The terminal 490 may display a vehicle exterior information output screen so that the user's individual consent to the damaged image can be processed. That is, information on the damaged image of the terminal 490 may be mapped to and outputted to a vehicle image, and the vehicle image may be rotated, enlarged, reduced, or moved by a user's manipulation. The terminal 490 may display a button for giving consent to a damaged part of the vehicle image, and when the user presses the button, it is determined that individual consent has been made, and the consent information is transmitted to the server 499. can be sent to

55(b) is a diagram for explaining a vehicle exterior photographing request screen among output screens of a shared vehicle management system.

Referring to (b) of FIG. 55 , a vehicle exterior photographing request screen according to an exemplary embodiment may include a guide unit and a photographing unit. The vehicle exterior photographing request screen may include a guide unit for guiding photographing so that main parts of the exterior of the vehicle may be photographed, and a photographing unit including a photographing button for photographing the exterior of the vehicle. In this case, the vehicle exterior photographing request screen according to an embodiment may be a screen that is necessarily output before the user uses the vehicle, but is not limited thereto. That is, the vehicle appearance photographing request screen may be selectively output by the user.

55(c) is a view for explaining a screen for outputting information on whether or not additional damage has occurred to the vehicle after photographing the exterior of the vehicle is completed.

Referring to (c) of FIG. 55 , the vehicle damage information screen according to an embodiment includes a damage information table, a selection window for selecting whether to confirm or agree to the occurrence of additional damage, and/or damage displaying additional damage information. Can contain images.

The screen for outputting additional damage information may include a damage information table displaying information on damage occurring to the shared vehicle. The damage information table may include damaged vehicle parts, damage types, damage levels, and the number of damages.

The screen for outputting the additional damage information may include a selection window for selecting whether or not to agree to the new damage information. The user can click the 'Confirm' button if he or she agrees with the information on the new damaged area (ND), and can click the 'Objection' button if he or she does not agree. When the 'Confirm' button is clicked by the user, the terminal 490 may inform the server 499 that the collective agreement on the existing damage information has been performed.

Alternatively, the terminal 490 may perform a procedure for obtaining individual consent for new damage information by using a plurality of individual buttons, similar to the above-described procedure for obtaining consent for the existing damage information.

A screen for outputting additional damage information may include a damage image on which additional damage information is displayed. The damaged image may include damaged area information, which may include an existing damaged area ED and a new damaged area ND. In this case, the existing damaged area ED and the new damaged area ND may be displayed differently. The existing damaged area ED and the new damaged area ND may be displayed as bounding boxes of different colors or shapes.

55(d) is a diagram for explaining a screen including detailed information on damage information generated in a shared vehicle.

Referring to (d) of FIG. 55, when the user clicks the 'Objection' button, the output screen of the shared vehicle management system is a damaged image extracted before the user uses the shared vehicle and a damaged image extracted after the user uses the shared vehicle. can be displayed. In this case, the damaged image extracted before using the shared vehicle and the damaged image extracted after using the shared vehicle may be damaged images detected in directions corresponding to each other. As another example, the damaged image extracted before using the shared vehicle and the damaged image extracted after using the shared vehicle include at least one main part of the vehicle, and at least one main part of the vehicle may be common. Meanwhile, the damaged image extracted before using the shared vehicle and the damaged image extracted after using the shared vehicle may be mapped onto a processed image, for example, a 3D image of the vehicle, rather than an actual photographed image.

4.3 Confirm Scenario

According to an embodiment, the mobile-based vehicle management process may include a step in which the vehicle return process is completed only when the shared vehicle user photographs the exterior of the shared vehicle after using the shared vehicle and consents to additional damage information.

In addition, the mobile-based vehicle management process may include allowing the vehicle to be used when the shared vehicle user agrees to the current appearance of the shared vehicle before using the shared vehicle.

In addition, in the mobile-based vehicle management process, if the shared vehicle user does not agree with the current exterior condition of the shared vehicle before using the shared vehicle, a step of re-photographing and registering the disagreed part or the entire exterior of the vehicle should be performed. It may include steps to enable use.

56 is a diagram schematically illustrating a process in which a plurality of shared vehicle users use a shared vehicle and end the use according to a time sequence.

Referring to FIG. 56 , a first user may start using a shared vehicle at a first time point T1 and end the use of a shared vehicle at a second time point T2 . In addition, the second user may start using the shared vehicle at the third time point T3 and end the use of the shared vehicle at the fourth time point T4. In this case, it may be assumed that the first user and the second user use the same shared vehicle. Also, the first to fourth points of view refer to different points of view, and the first point of view may be earlier than the third point of time.

Also, referring to FIG. 56 , the first user's time to use the shared vehicle is a first section (P1) between the first time point and the second time point, and the second user's time to use the shared vehicle is the third time point and the fourth time point. It is the third section P3 between. That is, the first period P1 may refer to a period in which the first user uses the vehicle, and the third period P3 may indicate a period in which the second user uses the vehicle.

Meanwhile, the period from the end of the first user's use of the shared vehicle to the start of the second user's use of the shared vehicle may mean a period in which the shared vehicle is parked without being operated by the user. That is, the second section P2, which is a time period between the second time point T2 and the third time point T3, may mean a section in which the shared vehicle is parked without being operated by the user.

Hereinafter, with reference to FIG. 56, the step of using the vehicle after the shared vehicle user consents to the existing damage information of the shared vehicle before using the shared vehicle, the step of using the vehicle after re-photographing the exterior of the vehicle if the user does not agree, and the step of using the shared vehicle after re-photographing the exterior of the shared vehicle user The step of taking a picture of the exterior of the shared vehicle after use and returning it will be described in detail.

4.3.1 After using the vehicle, take a picture of the vehicle exterior and manage additional damage

57 is a view for explaining a vehicle return process in which a shared vehicle user photographs the exterior of the vehicle after using the shared vehicle and checks additional damage information. Hereinafter, with reference to FIGS. 56 and 57 , a process of photographing the exterior of a vehicle and confirming additional damage information after a shared vehicle user uses a shared vehicle will be described.

Referring to FIG. 57 , the mobile-based vehicle management system may include a terminal 490 and a server 499. The terminal 490 may be the terminal 490 of the first user. The vehicle use time of the first user may end at a second point in time T2.

The terminal 490 of the first user may recognize that the vehicle use time has ended under the control of the first user, or the terminal 490 of the first user may recognize that the time for using the vehicle has ended, or based on information transmitted from the server 499. It may be displayed that the vehicle use time has ended.

The terminal 490 of the first user may display a shooting guide through the output unit 496 . Display of the shooting guide through the output unit 496 may be an optional process as described in 4.2.1.1.

The terminal 490 of the first user may capture a first image through the camera module 491 under the control of the first user. The first image may include at least one image related to the appearance of a vehicle used by the first user. The first image may be a plurality of images including main parts of the exterior of the vehicle. The first image may be an image taken at a point of time corresponding to the second point of view T2.

The terminal 490 of the second user may also capture a third image using the camera module 491 at a fourth time point T4 when the vehicle use time ends under the control of the second user. The third image may include at least one image related to the appearance of a vehicle used by the second user. The third image may be a plurality of images including main parts of the exterior of the vehicle. The third image may be an image captured at a point of view corresponding to the fourth point of view T4.

The terminal 490 of the first user may store the first image in the storage unit 493 and transmit the first image to the server 499 through the communication unit 494 . The terminal 490 of the first user may transmit the first image to the server 499 at a time point corresponding to the second time point T2. In addition, the terminal 490 of the second user may also transmit the third image to the server 499 at a time point corresponding to the fourth time point T4.

The server 499 may receive the first image from the first user terminal 490 .

The server 499 may determine whether the vehicle is additionally damaged based on the first image.

The server 499 may compare the first image with data stored in the server 499 to determine whether the shared vehicle is additionally damaged. The data stored in the server 499 may be existing damage information of the shared vehicle. The server 499 may analyze the first image to determine whether the shared vehicle is additionally damaged. That is, the server 499 may determine whether there is damage to the vehicle used by the shared vehicle user while the shared vehicle user is driving.

As a result, the server 499 compares the image of the shared vehicle after the end of use by the shared vehicle user with the existing damage information of the shared vehicle to identify additional damage information occurring to the vehicle, thereby determining who is responsible for damage to the shared vehicle. can be clarified, and it is possible to systematically manage damages occurring to shared vehicles in a time-series manner. Exemplarily, the server 499 compares the third image with the first image to determine additional damage information, and thus, from the second time point T2, which is the end point of use of the first user, to the end point of use of the second user, the second point in time T2. It is possible to identify and manage damage information generated between 4 points in time (T4). In other words, the server 499 compares the third image with the first image to determine additional damage information, thereby identifying and managing damage information generated in the second section P2 or the third section P3.

When the server 499 determines that no additional damage has occurred to the vehicle, the server 499 may update damage information on the corresponding vehicle. That is, when the server 499 determines that no additional damage has occurred to the vehicle, the damage information on the corresponding vehicle indicates that no additional damage has occurred to the corresponding vehicle and that the damage information on the corresponding vehicle is the same as the existing damage information. can be updated. In this case, the updated damage information may be defined as first information.

Exemplarily, when the server 499 determines that no additional damage has occurred to the vehicle from the first image or the third image captured at the second time point T2 or the fourth time point T4, the server 499 is damaged on the corresponding vehicle. Damage information on the corresponding vehicle may be updated so that the information is the same as the existing damage information.

As another example, when the server 499 determines that no additional damage has occurred to the vehicle from the third image captured at the fourth time point T4, the damage information on the corresponding vehicle is stored in the third image taken at the second time point T2. 1 It can be updated with the same content as the damage information determined from the image.

When determining that additional damage has occurred to the vehicle, the server 499 may transmit corresponding additional damage information to the user's terminal 490 .

The user's terminal 490 may receive additional damage information from the server 499 .

When the additional damage information is received, the terminal 490 of the user may confirm whether the user agrees to the additional damage.

The user's terminal 490 may indicate to the user whether to agree to the additional damage using the output unit 496, and allow the user to select whether to agree to the additional damage information. Through the input unit 495, the user may select whether or not to agree to whether the additional damage information is damage that occurred during the period of using the vehicle.

In this case, the additional damage information may include additional damage information generated during the current user's use period. Exemplarily, the additional damage information may refer to damage information additionally generated in the vehicle between a first time point T1, which is the start point of use by the first user, and a second point in time T2, which is the end point of use of the first user. can That is, the additional damage information may refer to damage information additionally generated in the vehicle in the first section P1, which is the usage period of the first user.

The additional damage information at the fourth point in time T4 is additionally generated in the vehicle between the third point in time T3, which is the start point of use by the second user, and the fourth point in time T4, which is the end point of use by the second user. It may mean damage information. That is, the additional damage information may refer to damage information additionally generated in the vehicle in the third period P3, which is the use period of the second user.

The additional damage information may refer to damage information additionally generated in the vehicle between the second time point T2, which is the end point of use by the first user, and the fourth point in time T4, which is the end point of use of the second user. That is, the additional damage information may refer to damage information additionally generated in the vehicle in the second section P2 or the third section P3.

In other words, when the terminal 490 receives the additional damage information from the server 499, the second user determines that the additional damage information is transmitted between the second time point T2 or the third time point T3 and the fourth time point T4. You can choose whether or not to agree to the damage caused to As another example, if the terminal 490 receives the additional damage information from the server 499, the second user agrees whether the additional damage information is damage generated in the second section P2 or the third section P3. can choose

The vehicle return process can be completed only after going through a procedure for checking the additional damage information by the user's terminal 490 . That is, when the time reserved by the user elapses, the use of the vehicle is terminated, but the vehicle return process can be completed only after the user agrees to or raises an objection to the additional damage information.

If the user agrees to the additional damage information, the vehicle return process can be completed. If the user acknowledges that he or she has no objection to the additional damage information, the vehicle return process may be completed.

Exemplarily, when the first user agrees to the additional damage information using the first user terminal 490, the vehicle return process may be completed.

When the second user uses the second user terminal 490, the additional damage information is damage that has additionally occurred to the vehicle between the second time point T2 or the third time point T3 and the fourth time point T4, or If it is agreed that the damage is additionally caused to the vehicle in the second section (P2) or the third section (P3), the vehicle return process may be completed.

If the user does not agree to the additional damage information, the user may file an objection regarding the reason for not agreeing to the additional damage information, and when the objection is completed, the vehicle return process may be completed. That is, if the user does not agree with all or part of the additional damage information, he or she may go through an objection procedure for the disagreement, and when the objection procedure is completed, the vehicle return process may be completed.

Exemplarily, the second user indicates that the additional damage information is damage that additionally occurred to the vehicle between the second time point T2 or the third time point T3 and the fourth time point T4, or the second period P2 or If you do not agree at all or partially that the damage is additionally caused to the vehicle in the third section (P3), you may file an objection for the part you do not agree with. Thereafter, when the objection procedure is completed, the vehicle return process may be completed.

When the vehicle return process is completed, the terminal 490 of the first user may reflect the user's consent to the additional damage information and transmit it to the server 499 .

In addition, the server 499 may update the vehicle damage information based on the additional damage information in which the consent of the first user received from the terminal 490 of the first user is reflected. In this case, the updated vehicle damage information may be defined as first information.

As such, the server 499 updates the vehicle damage information based on the information obtained from the terminal 490, thereby managing the damage information occurring on the exterior of the vehicle in a time-sequential manner, and also the damage information occurring on the exterior of the vehicle. It is possible to clarify where the responsibility is located so that repair, inspection and replacement cost claims can be made smoothly.

4.3.2 Whether or not to agree to the existing damage information before using the vehicle and re-photographing the exterior of the vehicle

According to an embodiment, the mobile-based vehicle management process may include enabling use of the vehicle when the shared vehicle user agrees to existing damage information of the vehicle before using the vehicle. In addition, in the mobile-based vehicle management process, if the shared vehicle user does not agree with all or part of the existing damage information of the vehicle before using the vehicle, a step of re-photographing the disagreeable part or the entire vehicle exterior should be performed before using the vehicle. It may include steps to make this possible.

58 is a view for explaining a vehicle use process in which a shared vehicle user indicates whether or not to agree to existing damage information of a shared vehicle before using the shared vehicle and, if not, takes a photograph of the exterior of the vehicle. Hereinafter, with reference to FIGS. 56 and 58, a series of steps that a shared vehicle user must perform before using a shared vehicle will be described.

Referring to FIG. 58 , the user's terminal 490 may activate the smart key before a predetermined time of using the vehicle. The server 499 can control the activation of the smart key of the terminal 490 of the user. The terminal 490 of the second user may activate the smart key before a predetermined time of the third point in time.

When the smart key is activated, the terminal 490 of the second user may request existing damage information on the corresponding vehicle from the server 499 .

In this case, the existing damage information may mean information on the current external state of the shared vehicle. In addition, the existing damage information may refer to damage information of a vehicle updated by a previous user who used the vehicle immediately before the current user uses the vehicle. Exemplarily, the existing damage information may mean damage information that has already occurred in the shared vehicle before the shared vehicle user uses the shared vehicle. As another example, the existing damage information may mean damage information generated to a shared vehicle by a previous shared vehicle user. As another example, the existing damage information may mean damage that occurred before the current shared vehicle user used the shared vehicle after the previous shared vehicle user completed returning the shared vehicle, for example, damage during parking.

Also, referring to FIGS. 56 and 58 , the existing damage information may refer to first information updated from a first image taken at a second point in time T2 after the first user uses the vehicle. For example, if the second user is defined as the current user and the first user is defined as the previous user, the existing damage information is stored between the first time point T1 and the second time point T2 or the third time point T3. It may mean damage information generated in , and the existing damage information may mean damage information generated in the first section (P1) or the second section (P2). Alternatively, when the first user does not perform any photographing, the existing damage information may be an image photographed by a user who photographed the most recent image.

Meanwhile, the existing damage information may be expressed through a real image of a shared vehicle or a processed 3D image. For example, existing damage information may be provided in a form in which a damaged part is displayed as a bounding box on an actual image of a shared vehicle. As another example, the existing damage information may be provided in a form in which a damaged part is displayed as a bounding box on a processed 3D image in which information about shared vehicles is reflected. In this case, when the existing damage information is provided in the form of a processed 3D image, the shared vehicle user can more conveniently grasp overall damage information on the exterior of the vehicle through a method of rotating the 3D image 360 degrees.

After receiving the existing damage information request message from the second user's terminal 490, the server 499 may retrieve the existing damage information and transmit the corresponding content to the second user's terminal 490. Thereafter, the second user's terminal 490 receives the existing damage information, displays the existing damage information on the second user's terminal 490, and then displays a message prompting the second user to check the vehicle exterior information. .

The second user may select whether to agree to the existing damage information. That is, the terminal 490 of the second user may display existing damage information occurring in the shared vehicle to the shared vehicle user, and then display a screen through which the second user can select whether to agree to the existing damage information. there is. The consent may include collective consent and partial consent for the entire damage information.

The terminal 490 of the second user of the mobile-based vehicle management process may activate the door open function and start function of the shared vehicle when the second user agrees to the existing damage information of the shared vehicle before using the shared vehicle. The terminal 490 of the second user may activate the door open function and start function of the shared vehicle when the second user inputs collective agreement through the terminal 490 . That is, the second user may use the shared vehicle when collectively agreeing to the existing damage information of the shared vehicle before using the shared vehicle.

In this case, that the second user agrees with the existing damage information of the vehicle may mean agreeing that the existing damage information displayed on the terminal 490 of the second user matches the current appearance of the vehicle.

Although not shown, when collective agreement is input by the second user, the terminal 490 of the second user may transmit information about collective agreement to the server 499 . The server 499 may transmit data so that the terminal 490 of the second user can activate the door opening and starting functions after storing information on collective agreement.

When collective consent is input by the second user, the server 499 may store information indicating that the exterior state of the vehicle is the same as before.

If the second user does not agree to the existing damage information before using the vehicle, the second user's terminal 490 may display the second user so that the current exterior state of the vehicle can be photographed and registered. At this time, the terminal 490 of the second user may provide a photographing guide so that the second user may photograph the current exterior of the vehicle before using the vehicle. Since the photographing guide overlaps with the above description in 4.2.1.1, a detailed description thereof will be omitted.

More specifically, referring to FIG. 56 , the second user's terminal 490 may display the second user's vehicle exterior photographing at a third time point T3 when the vehicle usage time starts. In this case, the vehicle image taken by the second user before use may be the second image. That is, an image photographed by the second user's terminal 490 at a time point corresponding to the third point of view T3 may be defined as a second image. The second image is an image of the exterior of the vehicle, and may include a plurality of images including main parts of the exterior of the vehicle. The first user may also photograph the exterior of the vehicle at the first point in time T1 when the vehicle use time starts.

The second user's terminal 490 proceeds with the external photographing procedure only when the second user does not agree with the existing damage information, thereby performing a confirmation process before using the shared vehicle when there is no dispute about the damage information. can be more simplified.

Image data photographed by the second user may be stored and registered in the terminal 490 of the second user and then transmitted to the server 499 . For example, the second image captured by the second user at the third point of view T3 may be stored in the terminal 490 of the second user and then transmitted to the server 499 . Also, an image captured by the first user at the first point of view T1 may be transmitted to the server 499 in the same way.

After receiving the second image from the terminal 490 of the second user, the server 499 may determine whether additional damage has occurred to the vehicle based on the second image and existing damage information. More specifically, after receiving the second image from the terminal 490 of the second user, the server 499 may compare the second image with existing damage information of the corresponding vehicle to determine whether additional damage has occurred to the vehicle. That is, the server 499 may determine whether the current appearance state of the vehicle is different from existing damage information.

Illustratively, the server 499 compares and analyzes the second image taken at a third time point T3 before use by the second user with the first image taken at a second time point T2 after use by the first user. It is possible to determine whether additional damage has occurred to the vehicle. That is, the server 499 may determine whether additional damage has occurred in the second period P2 between the second time point T2 and the third time point T3.

As a result, the server 499 may update information on the current appearance of the shared vehicle by determining whether additional damage occurs, identify damage information missing from the existing damage information, and terminate the use of the previous user. Afterwards, it is possible to provide an effect of identifying additional damage information generated during parking. Exemplarily, the server 499 compares the second image with the first image to determine additional damage information, and thus, from the second time point T2, which is the end point of use by the first user, to the second point in time, which is the start point of use by the second user. Damage information generated between the three points of time T3, that is, damage information generated in the second period P2 can be identified and managed.

Meanwhile, the second user may not agree to all or part of the existing damage information of the shared vehicle, and the extent of re-photographing the exterior of the shared vehicle may vary depending on whether the second user agrees in whole or in part. A detailed description of this will be described later.

When it is determined that additional damage has occurred to the vehicle, the server 499 may update damage information on the corresponding vehicle. That is, when it is determined that additional damage has occurred to the vehicle, the server 499 may update existing damage information to match the current appearance of the vehicle. At this time, the updated damage information may be defined as second information.

For example, when determining that additional damage has occurred to the vehicle from the second image captured at the third time point T3, the server 499 may update existing damage information to match the current appearance state of the vehicle. . The updated contents may relate to damage information missing from the existing damage information, or may relate to additional damage information generated during parking after the previous user's use has ended.

When the server 499 determines that no additional damage has occurred to the vehicle, the server 499 may transmit the determination result to the second user's terminal 490, and the second user's terminal 490 receives the determination from the server 499. Upon receiving the result, the second user can select whether or not to agree to the judgment result. In this case, the determination result may include information indicating that there is no problem with the existing damage information, that is, information indicating that the existing damage information is the same as the current appearance state of the vehicle.

The terminal 490 of the second user may activate the door open and start function of the vehicle when the second user completes the procedure for confirming the determination result.

The terminal 490 of the second user may activate the door open and start function of the vehicle when the second user agrees with the determination result.

Although not shown, the server 499 may control the door opening and starting functions of the vehicle to be activated when the second user's terminal 490 transmits the second image, that is, additional damage described above. Before proceeding with the determination and consent process, the terminal 490 of the second user may activate the door opening and starting functions of the vehicle. In this case, in the process of driving the vehicle, the server 499 may proceed with the additional damage determination and consent procedure, and proceed with the additional damage determination and consent procedure until the vehicle operation is terminated, and after the additional damage determination and consent procedure is completed, the vehicle The return process can be terminated. Alternatively, the server 499 may proceed with an additional damage judgment and agreement procedure after the vehicle return procedure. The server 499 controls the door opening and starting functions of the vehicle to be activated when the second user's terminal 490 transmits the second image, thereby simplifying procedures before starting vehicle operation. there is.

When consent to the determination result is input by the second user, the server 499 may store information indicating that the exterior state of the vehicle is the same as before.

In addition, when the second user does not agree with the determination result, the terminal 490 of the second user may display a reason why the second user does not agree with the determination result so as to raise an objection. Thereafter, the terminal 490 of the second user may activate the door open and start function of the vehicle when the second user's objection is completed.

When an objection to the determination result is inputted by the second user, the server 499 may store information indicating that the external state of the vehicle is the same as before or additional damage has occurred according to the objection result. In other words, the server 499 may store information that additional damage has occurred to the exterior state of the vehicle when the objection to the judgment result input by the second user is accepted, and the objection is not accepted. If not, information that the exterior state of the vehicle is the same as before may be stored.

In addition, the server 499 may update vehicle damage information based on the existing damage information in which the user's consent and/or objection information received from the terminal 490 are reflected. In this case, the updated vehicle damage information may be defined as second information.

Exemplarily, the second user's terminal 490, when the second user agrees that the existing damage information corresponds to the second image captured at the third point in time T3, which is the second user's use start point, vehicle may activate the door open and start function of the vehicle and, if you do not agree, activate the door open and start function of the vehicle upon completion of the appeals process.

Although not shown, the server 499 may control the door opening and starting functions of the vehicle to be activated after the objection procedure by the terminal 490 of the second user and the processing procedure are completed, that is, The server 499 may activate the door opening and starting functions of the vehicle after the objection procedure by the terminal 490 of the second user and all consent procedures are completed. The server 499 controls the door opening and starting functions of the vehicle to be activated after the objection procedure by the terminal 490 of the second user and the processing procedure are completed, thereby preventing damage to the vehicle that may occur later and There is an effect of preventing related disputes. In this way, the server 499 updates the vehicle damage information based on the information obtained from the terminal 490, thereby managing damage information occurring on the exterior of the vehicle in a time-sequential manner. In addition, by clarifying the location of responsibility for damage information occurring on the exterior of the vehicle, it is possible to smoothly claim repair, inspection, and replacement costs.

In addition, the server 499 updates the vehicle damage information based on the information obtained from the terminal 490, so that the shared vehicle user clarifies the damage information that has already occurred before using the vehicle, so that the existing damage to the vehicle is determined by the server 499. It can be made clear that there is no liability. Furthermore, the server 499 updates the vehicle damage information based on the information obtained from the terminal 490, so that the shared vehicle user can only take responsibility for additional damage while using his/her vehicle, and the manager can also take responsibility for damage caused to the vehicle. Responsibilities can be clarified.

In addition, the server 499 updates the vehicle damage information based on the information obtained from the terminal 490, so that the manager can grasp the damage information generated when the vehicle is parked without being used by the user, and respond accordingly. action may be possible. For example, referring to FIG. 56 , the server 499 updates the vehicle damage information based on the information obtained from the terminal 490, so that the administrator can determine the second time point T2, which is the end point of use of the first user, and the second time point T2. 2 Damage information generated between the third time point T3, which is the user's starting point of use, that is, damage information generated in the second period P2 can be grasped, and corresponding responses and measures may be possible.

Meanwhile, as described above, the shared vehicle user may not agree with all or part of the current appearance of the shared vehicle. That is, the shared vehicle user may partially indicate whether or not to agree to the existing damage information of the shared vehicle.

59 is a diagram for explaining a process of re-photographing the exterior of a shared vehicle when a shared vehicle user does not agree with existing damage information of the shared vehicle. Hereinafter, a process of re-photographing the exterior of a shared vehicle when a shared vehicle user does not agree with existing damage information of the shared vehicle will be described with reference to FIG. 59 .

Referring to (a) of FIG. 59 , when the second user does not agree on all existing damage information of the shared vehicle, the second user's terminal 490 displays the entire exterior of the shared vehicle so that re-photographing can be performed. can do. At this time, the second user's terminal 490 may output a recapture guide that guides all major parts of the exterior of the vehicle to be photographed. The recapture guide may be provided based on information input by the second user when making a reservation or information obtained through a learned neural network model.

Referring to (b) of FIG. 59 , when the second user does not agree with some of the existing damage information of the shared vehicle, the terminal 490 of the second user guides re-photographing of the parts that do not agree. can The terminal 490 of the second user may display a recapture guide corresponding to a part to which the second user does not agree. The second user proceeds to take a picture according to the retake guide, so that the second user terminal 490 can obtain an image for a part that does not agree. The re-photographing guide may be provided based on information input by the shared vehicle user when making a reservation or information obtained through a learned neural network model. Through this process, the terminal 490 of the second user may store an image of a part that the user disagrees with and transmit the image to the server 499 . The terminal 490 of the second user can simplify the re-photographing process by controlling re-photographing only for the part to which the second user disagrees.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (15)

A vehicle management assistance device for obtaining vehicle information related to the exterior of a vehicle based on a vehicle image,
an image acquiring unit acquiring at least one image related to the vehicle;
an additional information obtaining unit acquiring additional information related to the characteristics of the vehicle; and
A processor for obtaining the vehicle information by using a neural network model learned to obtain the vehicle information based on at least one image related to the vehicle and additional information related to characteristics of the vehicle; including,
The neural network model is trained to obtain a first feature set based on at least one image associated with the vehicle and includes a first portion comprising at least one convolutional neural network layer, and a first feature set obtained from the first portion. and a second part learned to obtain a second set of features based on additional information related to the characteristics of the vehicle;
The processing unit obtains a target vehicle image obtained by photographing the exterior of the target vehicle through the image acquisition unit,
Obtaining target additional information related to the characteristics of the target vehicle through the additional information acquisition unit;
Based on the target vehicle image, a first target feature set corresponding to the target vehicle obtained from the first part of the neural network model is obtained, wherein the first target feature set is a part, type, and color of the target vehicle. , including at least one first feature value indicating a configuration and shape,
obtaining a second target feature set corresponding to the target vehicle from the second part of the neural network model based on the obtained first target feature set corresponding to the target vehicle and the target additional information; The additional information may include first additional information determined based on non-image data including a Vehicle Identification Number (VIN) of the target vehicle, and second determined based on a vehicle interior image obtained by photographing the inside of the target vehicle. 2 additional information, the second target feature set includes at least one second feature value related to whether the target vehicle has external damage and the external damage type of the vehicle, and the second target feature set includes the target vehicle A first region feature set representing a region in which a first part constituting the exterior of the vehicle is distributed and a second region feature set representing a region in which a second component is distributed, wherein the region in which the first part is distributed and the first region feature set are included. The areas where the two parts are distributed are different from each other,
Obtaining target vehicle information about the target vehicle based on the obtained second target feature set corresponding to the target vehicle;
Fleet management aids.
According to claim 1,
The neural network model is learned to acquire damage information related to external damage of the vehicle, and the second feature set includes at least one feature value related to whether the vehicle is damaged or not.
Fleet management aids.
According to claim 1,
The neural network model is trained to acquire part information related to a part constituting the exterior of the vehicle, and the second feature set includes at least one feature value related to a part constituting the exterior of the vehicle.
Fleet management aids.
delete delete delete According to claim 1,
the processing unit obtains, based on the first feature set and the additional information from the second part, the second feature set associated with at least one damaged region located outside the target vehicle;
Wherein the second feature set comprises a damage region feature set indicating a location of the at least one damaged region.
Fleet management aids.
Based on the image acquisition unit for acquiring at least one image related to the vehicle, the additional information acquisition unit for acquiring additional information related to the characteristics of the vehicle, and the at least one image related to the vehicle and the additional information related to the characteristics of the vehicle In the vehicle management assistance method of a vehicle management assistance device including a processing unit for obtaining vehicle information related to the exterior of the vehicle,
obtaining a target vehicle image acquired by photographing the outside of the target vehicle through the image acquisition unit;
obtaining target additional information related to characteristics of the target vehicle through the additional information obtaining unit; and
obtaining target vehicle information about the target vehicle based on the additional target information related to the characteristics of the target vehicle and the obtained target vehicle image of the target vehicle; Including,
The obtaining of the target vehicle information about the target vehicle is performed using a neural network model trained to obtain the vehicle information based on at least one image related to the vehicle and additional information related to characteristics of the vehicle, and ,
Obtaining the target vehicle information on the target vehicle,
obtaining a first target feature set corresponding to the target vehicle obtained from a first part of the neural network model based on the target vehicle image, wherein the first target feature set includes parts, types, and colors of the target vehicle , including at least one first feature value indicating a configuration and shape,
obtaining a second target feature set corresponding to the target vehicle from a second part of the neural network model based on the obtained first target feature set corresponding to the target vehicle and the target additional information; The additional information may include first additional information determined based on non-image data including a Vehicle Identification Number (VIN) of the target vehicle, and second determined based on a vehicle interior image obtained by photographing the inside of the target vehicle. 2 additional information, the second target feature set includes at least one feature value related to whether the target vehicle has external damage and the external damage type of the vehicle, and the second target feature set includes A first region feature set representing a region in which the first component constituting the exterior is distributed and a second region feature set representing a region in which the second component is distributed, wherein the region in which the first component is distributed and the second component are included. The regions in which these distributions are different from each other, and
Acquiring target vehicle information about the target vehicle based on the obtained second target feature set corresponding to the target vehicle.
A vehicle management aid method.
According to claim 8,
The neural network model is learned to acquire damage information related to external damage of the vehicle, and the second feature set includes at least one feature value related to whether the vehicle is damaged or not.
A vehicle management aid method.
According to claim 8,
The neural network model is trained to acquire part information related to a part constituting the exterior of the vehicle, and the second feature set includes at least one feature value related to a part constituting the exterior of the vehicle.
A vehicle management aid method.
delete delete delete According to claim 8,
Based on the first feature set and the additional information from the second part, the second feature set associated with at least one damaged area located outside the target vehicle is obtained;
Wherein the second feature set comprises a damage region feature set indicating a location of the at least one damaged region.
A vehicle management aid method.
A computer-readable recording medium storing a program for performing the method according to any one of claims 8 to 10 and 14.

Images