CN113408345B - Vehicle appearance cleanliness judgment method and device, server and storage medium - Google Patents
Vehicle appearance cleanliness judgment method and device, server and storage medium Download PDFInfo
- Publication number
- CN113408345B CN113408345B CN202110524024.7A CN202110524024A CN113408345B CN 113408345 B CN113408345 B CN 113408345B CN 202110524024 A CN202110524024 A CN 202110524024A CN 113408345 B CN113408345 B CN 113408345B
- Authority
- CN
- China
- Prior art keywords
- vehicle
- vehicle appearance
- images
- cleanliness
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000003749 cleanliness Effects 0.000 title claims abstract description 100
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000000605 extraction Methods 0.000 claims abstract description 17
- 238000001514 detection method Methods 0.000 claims description 22
- 230000011218 segmentation Effects 0.000 claims description 6
- 230000007613 environmental effect Effects 0.000 claims description 4
- 230000004048 modification Effects 0.000 claims description 4
- 238000012986 modification Methods 0.000 claims description 4
- 239000013598 vector Substances 0.000 description 19
- 238000004891 communication Methods 0.000 description 17
- 230000006870 function Effects 0.000 description 10
- 230000008569 process Effects 0.000 description 9
- 238000012545 processing Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 230000002708 enhancing effect Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000036651 mood Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
- 238000010897 surface acoustic wave method Methods 0.000 description 1
- 230000029305 taxis Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention relates to a method and a device for judging the appearance cleanliness of a vehicle, a server and a storage medium, relating to the technical field of Internet of things and comprising the following steps: splicing the plurality of vehicle appearance images according to the shooting angle for shooting each vehicle appearance image, and segmenting the spliced images to obtain a plurality of segmented areas; performing feature extraction on each segmented region to obtain an image feature value corresponding to each segmented region, and determining a target image feature value according to the image feature value corresponding to each segmented region; and if the plurality of vehicle appearance images are determined to be the images of the same vehicle appearance shot at a plurality of shooting angles according to the target image characteristic value, determining the cleanliness of the vehicles in the plurality of vehicle appearance images. According to the embodiment of the invention, the plurality of vehicle appearance images can be judged to be the images of the same vehicle appearance shot by a plurality of shooting angles, so that the cleanliness is judged, and the accuracy of cleanliness judgment is improved.
Description
Technical Field
The invention relates to the technical field of Internet of things, in particular to a method and a device for judging the appearance cleanliness of a vehicle, a server and a storage medium.
Background
Automobiles provide the majority of transportation, for example, net appointments and traditional taxis. In the field of online taxi booking and traveling, a driver generally checks cards and dispatches the taxi, and is not much aware of whether the appearance of the taxi is clean or not. Or after a longer order pickup time, the vehicle tends to become messy. In such a state, the cleanliness of the appearance of the networked reservation vehicle directly influences the riding experience of passengers, including the passenger's will to get on the vehicle, the riding mood and the evaluation of the order completion. It is therefore necessary to check the cleanliness for the vehicle.
In the detection, the cleanliness of the whole vehicle body of the vehicle is detected, for example, the cleanliness needs to be detected in the places such as the lamp, the windshield, the door, the wheel, the roof, the tail and the like. However, some drivers often transmit clean images of other vehicles or transmit only the clean side of their own vehicle in order to be able to detect the cleanliness, which results in erroneous cleanliness detection and judgment.
Disclosure of Invention
The invention provides a method, a device, a server and a storage medium for judging the cleanliness of vehicle appearance, which can judge whether a plurality of vehicle appearance images comprise the same vehicle and a plurality of vehicle body characteristics of the vehicle so as to judge the cleanliness, and solve the problem that in the prior art, a driver transmits a clean image of other vehicles or only transmits one clean side of the vehicle to cause wrong cleanliness detection and judgment.
In a first aspect, a method for determining cleanliness of an exterior of a vehicle according to an embodiment of the present invention includes:
segmenting a plurality of vehicle appearance images to obtain a plurality of segmented regions;
performing feature extraction on each segmented region to obtain an image feature value corresponding to each segmented region, and determining a target image feature value according to the image feature value corresponding to each segmented region;
and if the plurality of vehicle appearance images are determined to be the images of the same vehicle appearance shot at a plurality of shooting angles according to the target image characteristic value, determining the cleanliness of the vehicles in the plurality of vehicle appearance images according to the plurality of vehicle appearance images.
According to the method, the multiple images for detecting the cleanliness can be segmented to obtain the multiple segmented areas, then the characteristic extraction is carried out on each area to obtain the corresponding image characteristic, the target image characteristic value is determined from the image characteristic, the multiple vehicle appearance images are judged to be the images of the same vehicle appearance shot at multiple shooting angles, if yes, the fact that one of the multiple vehicle appearance images is not the image of other vehicles, meanwhile, the clean side of the vehicle is not transmitted, the vehicle cleanliness detection is carried out on the qualified images, and the accuracy of the vehicle cleanliness detection is improved.
In one possible implementation, the method for segmenting a plurality of vehicle appearance images to obtain a plurality of segmented regions includes:
and splicing the plurality of vehicle appearance images according to the shooting angle of each vehicle appearance image to obtain a spliced image, and segmenting the spliced image to obtain a plurality of segmented areas.
According to the method, the plurality of appearance images can be spliced according to the shooting angle of each vehicle appearance image, and when the images are segmented again, the segmented areas can possibly comprise the images of continuous scenes in two or even a plurality of vehicle appearance images, so that when the features are extracted, the extraction of a plurality of features from the same scene can be avoided, and the processing speed is improved.
In one possible implementation, before segmenting the plurality of vehicle appearance images into a plurality of segmented regions, the method further includes:
receiving a departure request sent by a network car booking driver;
after determining the cleanliness of the vehicles included in the plurality of vehicle appearance images according to the plurality of vehicle appearance images, the method further includes:
if the cleanliness is larger than a preset cleanliness threshold value, the order receiving state of the vehicle driven by the network appointment vehicle driver is modified into a normal order receiving state; or
And if the cleanliness is not greater than the preset cleanliness threshold value, keeping the order receiving state of the vehicle driven by the network car booking driver unchanged, and informing the network car booking driver that the cleanliness of the vehicle driven by the network car booking driver is unqualified.
According to the method, the vehicle cleanliness detection process can be added in the departure process, so that the departure of vehicles with unqualified cleanliness can be avoided, and the vehicle using experience of passengers is improved.
In one possible implementation manner, before the segmenting the plurality of vehicle appearance images to obtain a plurality of segmented regions, the method further includes:
determining that the time difference between the acquisition time of the plurality of vehicle appearance images and the time of receiving the departure request is within a preset time length; and/or
And determining that the shooting positions of the plurality of vehicle appearance images and the current position of the vehicle driven by the online car booking driver are within a preset distance range.
The method can determine that the time difference between the acquisition time of the appearance images of the vehicles and the time of receiving the departure request is within the preset time length, so that the appearance images of the vehicles can be determined to be shot in real time instead of being obtained from a gallery, and the authenticity of the images is improved.
In a second aspect, an embodiment of the present invention provides a vehicle appearance cleanliness determination device, including:
the segmentation module is used for segmenting the plurality of vehicle appearance images to obtain a plurality of segmented areas;
the extraction module is used for extracting features of each segmented region to obtain an image feature value corresponding to each segmented region, and determining a target image feature value according to the image feature value corresponding to each segmented region;
and the detection module is used for determining the cleanliness of the vehicles in the multiple vehicle appearance images according to the multiple vehicle appearance images if the multiple vehicle appearance images are determined to be the images of the same vehicle appearance shot at multiple shooting angles according to the target image characteristic value.
In a possible implementation manner, the segmentation module is specifically configured to:
and splicing the plurality of vehicle appearance images according to the shooting angle of each vehicle appearance image to obtain a spliced image, and segmenting the spliced image to obtain a plurality of segmented areas.
In one possible implementation, the apparatus further includes:
the receiving module is used for receiving a departure request sent by a network car booking driver;
the state modification module is used for modifying the order receiving state of the vehicle driven by the vehicle ordering driver into a normal order receiving state if the cleanliness is larger than a preset cleanliness threshold; or
And if the cleanliness is not greater than the preset cleanliness threshold value, keeping the order receiving state of the vehicle driven by the network car booking driver unchanged, and informing the network car booking driver that the cleanliness of the vehicle driven by the network car booking driver is unqualified.
In one possible implementation, the apparatus further includes:
the determining module is used for determining that the time difference between the acquisition time of the plurality of vehicle appearance images and the time of receiving the departure request is within a preset time length; and/or
And determining that the shooting positions of the plurality of vehicle appearance images and the current position of the vehicle driven by the taxi appointment driver are within a preset distance range.
In a third aspect, an embodiment of the present invention provides a server, including:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the vehicle appearance cleanliness determination as defined in any one of the first aspects.
In a fourth aspect, the present application further provides a storage medium, wherein instructions when executed by a processor of a server enable the processor to perform the vehicle appearance cleanliness determination as set forth in any one of the embodiments of the first aspect.
In addition, for technical effects brought by any one implementation manner of the second aspect to the fourth aspect, reference may be made to technical effects brought by different implementation manners of the first aspect, and details are not described here.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention and are not to be construed as limiting the invention.
Fig. 1 is a schematic flow chart of a method for determining cleanliness of an exterior of a vehicle according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a vehicle appearance image segmentation provided by an embodiment of the invention;
FIG. 3 is a schematic diagram of a vehicle appearance image segmented to obtain a classification result according to an embodiment of the present invention;
FIG. 4 is a flow chart of a departure provided by an embodiment of the present invention;
fig. 5 is a structural diagram of a vehicle appearance cleanliness determination apparatus according to an embodiment of the present invention;
fig. 6 is a block diagram of a server according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.
The application scenario described in the embodiment of the present invention is for more clearly illustrating the technical solution of the embodiment of the present invention, and does not form a limitation on the technical solution provided in the embodiment of the present invention, and it can be known by a person skilled in the art that with the occurrence of a new application scenario, the technical solution provided in the embodiment of the present invention is also applicable to similar technical problems. Wherein, in the description of the present invention, unless otherwise indicated, "a plurality" means.
At present, when the cleanliness of a vehicle is detected, each part of the vehicle body needs to be detected, and the cleanliness of the vehicle is evaluated from the whole body, however, some drivers often transmit clean images of other vehicles or only transmit one clean side of the driver in order to be able to detect the cleanliness, so that the cleanliness detection judgment is wrong.
Therefore, according to the scheme of the embodiment of the invention, through detecting whether the image needing to be subjected to the cleanliness detection contains the same vehicle and contains a plurality of characteristics of the vehicle body of the vehicle, if so, the cleanliness detection is carried out, so that the condition that a driver uploads the image of other vehicles can be avoided, meanwhile, the condition that only the clean side of the vehicle is transmitted is avoided, and the accuracy of the vehicle cleanliness detection is improved.
The following describes specific embodiments of the present invention with reference to the drawings.
Referring to fig. 1, an embodiment of the present invention provides a method for determining cleanliness of an exterior of a vehicle, including:
firstly, segmenting a plurality of vehicle appearance images to obtain a plurality of segmented regions;
the plurality of vehicle appearance images described herein are vehicle appearance images for cleanliness detection.
As shown in fig. 2, for example, a single vehicle appearance image is divided into 15 regions, and the 15 regions after division do not have an intersection, that is, a plurality of divided regions may be from different portions of the same vehicle appearance image or from different vehicle appearance images. Therefore, the repeated extraction of the same part in the subsequent feature extraction can be avoided, and the workload is increased.
The method for dividing a plurality of vehicle appearance images to obtain a plurality of divided areas comprises the following steps:
s100: and splicing the plurality of vehicle appearance images according to the shooting angle of each vehicle appearance image to obtain a spliced image, and segmenting the spliced image to obtain a plurality of segmented areas.
Specifically, a plurality of vehicle appearance images are firstly zoomed, spliced and then cut, the segmentation operation can be skillfully selected according to the zoomed image size, and one area can not just occupy two pictures.
When a plurality of vehicle appearance images are stitched based on the shooting angle for shooting each vehicle appearance image, for example, when the shooting order is from the front of the vehicle, the left side of the vehicle, the back of the vehicle, and the right side of the vehicle, the images in front of the vehicle, the images on the left side of the vehicle, the images behind the vehicle, and the images on the right side of the vehicle are stitched in order, a large image is combined, and then the images are divided, and the features of the divided images are extracted.
S101: performing feature extraction on each segmented region to obtain an image feature value corresponding to each segmented region, and determining a target image feature value according to the image feature value corresponding to each segmented region;
referring to fig. 2, one vehicle appearance image includes a vehicle for cleanliness detection and a background portion, which are sorted from left to right once and sequentially from top to bottom, and a total of 15 areas, and when extracting features, 15 areas obtain image feature values corresponding to the 15 areas. It can be seen that the first to sixth regions, the tenth region, that is, the five regions in the uppermost row, and the leftmost region in the middle row and the rightmost region in the middle row are background portions. The three areas in the middle row from the leftmost area, that is, the seventh area to the ninth area, and all the areas in the lowermost row are areas of the vehicle subjected to the cleanliness detection.
The target image characteristic value comprises an image characteristic value representing the vehicle body and an image characteristic value representing the environment. The vehicle body refers to the shape and style of a wheel, the shape and style of a lamp, the style of a vehicle body, the shape and style of a handle, and the like.
When the target image characteristic value comprises an image characteristic value representing a vehicle body, in the characteristic extraction process, the target image characteristic value is determined from the image characteristic values corresponding to the 15 areas, namely the seventh area to the ninth area and the image characteristic values corresponding to all the areas in the bottom row.
The target image feature values may also include image feature values characterizing the vehicle body and image feature values characterizing the environment. For example, when a driver photographs a vehicle, the same environment may appear in a plurality of vehicle appearance images photographed in succession, and when a plurality of vehicle appearance images are subsequently analyzed as images of the same vehicle appearance photographed at a plurality of photographing angles, the same environment may be used for an auxiliary determination, so that the target image feature value includes an image feature value representing a vehicle body and an image feature value representing an environment at this time.
S102: and if the plurality of vehicle appearance images are determined to be the images of the same vehicle appearance shot at a plurality of shooting angles according to the target image characteristic value, determining the cleanliness of the vehicles in the plurality of vehicle appearance images according to the plurality of vehicle appearance images.
In order to enable the cleanliness detection, when the driver sees a stain on the left side of the own vehicle body, the driver may upload an image on the left side of another vehicle together with an image on another side of the own vehicle, for example, upload an image on the left side of another vehicle, an image on the rear side of the own vehicle, an image on the front side of the own vehicle, and an image on the right side of the own vehicle. In order to avoid the phenomenon, the invention provides that whether the plurality of vehicle appearance images are the images of the same vehicle appearance shot at a plurality of shooting angles is judged according to the target image characteristic value, the environmental characteristics of the vehicle can be considered, if the same environmental characteristics exist, the images shot in the same environment when the vehicle appearance images are shot are explained, the image characteristic value representing the vehicle body is considered, if the image characteristic value of the same vehicle exists, namely, the shape and the style of the same wheel, the shape and the style of the same lamp, the style of the same vehicle body, the shape and the style of the same handlebar and the like, the plurality of vehicle appearance images are considered to be the images of the same vehicle appearance shot at a plurality of shooting angles.
Meanwhile, in order to avoid the phenomenon that a driver does not upload a contaminated area on the vehicle body for the sake of cleanliness detection, and only uploads an image behind the vehicle, an image in front of the vehicle and an image on the right side of the vehicle when the left side of the vehicle body is contaminated, so that the cleanliness detection of the vehicle does not achieve the purpose of detection.
Through the two judgment processes, the two situations can be avoided, the images of the same vehicle appearance shot by a plurality of shooting angles are determined from a plurality of vehicle appearance images, and the cleanliness of the vehicles in the plurality of vehicle appearance images is determined according to the plurality of vehicle appearance images.
Step 101 is implemented by a transform Encoder model, where the transform model includes an Encoder portion and a decoder portion, and in the scheme of the present invention, only the Encoder portion of the transform model is used, and the portion mainly performs feature extraction on each segmented region to obtain an image feature value corresponding to each segmented region.
In the actual calculation process, the plurality of divided areas are input into the Transformer Encoder model together, before the plurality of divided areas are input into the Transformer Encoder model, one-dimensional vectors are added in front of the plurality of divided areas, and the added one-dimensional vectors are target image characteristic values determined according to the image characteristic values corresponding to each divided area. And then judging whether the plurality of vehicle appearance images are the images of the same vehicle appearance shot at a plurality of shooting angles by using the one-dimensional vector.
Further, the divided image may be subjected to the flattening processing, that is, the divided image may be represented by X × Y, X represents the number of pixels in the abscissa direction, and Y represents the number of pixels in the ordinate direction, and when the divided image is subjected to the flattening processing, X is changed to 1, and the number of pixels in the ordinate is X × Y. And (3) flattening each segmented image, splicing a one-dimensional vector in front, and inputting the vector into a Transformer Encoder model.
Meanwhile, because the multiple vehicle appearance images are spliced and then segmented, and then the transform Encoder model is adopted to extract the characteristics, the overlapping similarity exists at the continuous edges of the multiple vehicle appearance images, so the characteristics are extracted based on the transform Encoder model, and the relationship among the images can be effectively captured.
Furthermore, whether the plurality of vehicle appearance images are images of the same vehicle appearance shot at a plurality of shooting angles is judged according to the target image characteristic value through an MLP model, wherein the MLP model is a multilayer perception model and is actually a classifier, and whether the plurality of vehicle appearance images are images of the same vehicle appearance shot at a plurality of shooting angles can be distinguished.
Specifically, the one-dimensional vector obtained in the transform Encoder model is input to the MLP model, and the result of whether or not the plurality of vehicle appearance images are images of the same vehicle appearance captured at a plurality of capturing angles, that is, yes or no, is obtained.
The MLP model mainly comprises a LayerNorm layer and a full connection layer, the activation function is GELU, and a final classification result is obtained by inputting one-dimensional vectors into the MLP model. The LayerNorm layer is mainly used for normalizing data, so that unstable convergence caused by too large data difference is avoided; the fully connected layer mainly maps the learned distributed feature representation to a sample mark space to play a role of a classifier; the activation function of the GELU is decreased more quickly and more effectively than the activation function of the RELU.
In connection with the above description, an example is listed:
combining with fig. 3, four vehicle appearance images are spliced to form a large map, the size of which can be represented as H × 4W × C, where H represents pixels in the ordinate direction, W represents pixels in the abscissa direction, and C represents the number of channels, and then the large map is divided, for example, N regions are divided, the pixels of each region are represented as X × Y, X represents the number of pixels in the abscissa direction, and Y represents the number of pixels in the ordinate direction, where the divided regions are generally square, i.e., X ═ Y, a block-flattening operation is performed on the divided regions, assuming that the size of each region is 1 × Y, i.e., the abscissa is 1 pixel, and the number of pixels in the ordinate direction is X ═ Y, a one-dimensional vector is added to the front of N × 1 (X ×) and the added vector is input to a Transformer model, and performing feature extraction to obtain a plurality of target image feature values to form a one-dimensional vector, inputting the one-dimensional vector into an MLP model, judging whether the four vehicle appearance images are images of the same vehicle appearance shot at a plurality of shooting angles, obtaining a classification result, and if so, determining the cleanliness of the vehicle included in the four vehicle appearance images according to the four vehicle appearance images.
During actual operation, a Transformer Encoder model and an MLP model can be combined into an identification network, a plurality of vehicle appearance images are segmented to obtain a plurality of segmented regions, then each segmented region is subjected to leveling operation and is spliced into a vector, a one-dimensional vector is added into one vector and is input into the identification network, the identification network performs feature extraction on each segmented region to obtain an image feature value corresponding to each segmented region, and a target image feature value is determined according to the image feature value corresponding to each segmented region; and judging whether the plurality of vehicle appearance images are images of the same vehicle appearance shot at a plurality of shooting angles according to the target image characteristic value.
The method comprises the steps that a basic network is formed by a Transformer Encoder model and an MLP model, during training, a sample image is used as input and is input into the basic network formed by the Transformer Encoder model and the MLP model, and whether the sample image is an image of the same vehicle appearance shot by a plurality of shooting angles or not is used as an output result to conduct multi-round training to obtain an identification network. And judging whether the plurality of vehicle appearance images are images of the same vehicle appearance shot at a plurality of shooting angles through the identification network.
During training, a blank one-dimensional vector is added to the vector input to the basic network, so that the trained network can know which image characteristic values represent the vehicle body and the environment, and the image characteristic values are added to the one-dimensional vector, so that all the values in the one-dimensional vector are the image characteristic values contributing to the classification result.
For a sample image, collecting a sample and establishing a training data set;
in order to better detect the vehicle appearance image, a transverse shot picture is forced, namely the image is wider than higher.
Because the image resolution of the pictures shot by the mobile phone is different, the pictures are uniformly reduced by H W C size (H < W), H represents the height of the pictures, W represents the width of the pictures, and C represents the number of image channels. The four sample data of each vehicle do not necessarily need to be continuous clockwise or anticlockwise, and only four pictures of the same vehicle, namely the front, the back, the left and the right.
In order to improve the robustness of the trained network, sample data may be enhanced, specifically:
(1) enhancing positive sample data;
the same positive samples (four pictures of the same vehicle in front, back, left and right) are arranged and combined, and 24 sequences are provided in total, namely one positive sample is enhanced into 24 positive samples.
(2) Establishing and enhancing a negative sample data set;
the negative sample data set of the same vehicle picture is collected from four pictures of the same positive sample by a method of repeating one picture, two pictures, three pictures and four pictures respectively, and the four pictures are combined into a negative sample;
randomly taking four parts of negative sample data sets of different vehicle pictures from the positive sample data set, wherein one part is taken for each part;
mixing negative samples, respectively removing one and two positive samples, respectively randomly selecting one and two positive samples from the rest positive sample data sets, and supplementing the selected one and two positive samples, wherein one negative sample contains more than two pictures of the same vehicle and also contains data of different vehicles;
(3) enhancing a random sample;
the situation that the transverse picture is uploaded by a driver is reversed from top to bottom, and the sample data set is randomly turned over from top to bottom for the universality of data.
In the embodiment of the present invention, when the cleanliness detection is applied to a departure request, as shown in fig. 4, an embodiment of the present invention provides a departure flow, including:
s400: receiving a departure request sent by a network car booking driver;
s401: splicing the multiple vehicle appearance images according to the shooting angle of each vehicle appearance image to obtain spliced images, and segmenting the spliced images to obtain a plurality of segmented areas;
s402: performing feature extraction on each segmented region to obtain an image feature value corresponding to each segmented region, and determining a target image feature value according to the image feature value corresponding to each segmented region;
s403: judging whether the plurality of vehicle appearance images are images of the same vehicle appearance shot at a plurality of shooting angles according to the characteristic value of the target image, if so, executing S404, otherwise, executing S405;
s404: determining the cleanliness of the vehicles included in the plurality of vehicle appearance images according to the plurality of vehicle appearance images;
s405: informing the network that the image uploaded by the car booking driver is unqualified;
s406: if the cleanliness is larger than the preset cleanliness threshold value, the order receiving state of the vehicle driven by the network appointment vehicle driver is modified into a normal order receiving state;
s407: and if the cleanliness is not greater than the preset cleanliness threshold value, keeping the order receiving state of the vehicle driven by the network car booking driver unchanged, and informing the network car booking driver that the cleanliness of the vehicle driven by the network car booking driver is unqualified.
Therefore, the net appointment vehicle cleaning degree of the vehicle running on the road is qualified, and the image of the net appointment vehicle can be maintained.
Further, before the dividing the plurality of vehicle appearance images into the plurality of divided regions, the method further includes:
determining that the time difference between the acquisition time of the plurality of vehicle appearance images and the time of receiving the departure request is within a preset time length; and/or
And determining that the shooting positions of the plurality of vehicle appearance images and the current position of the vehicle driven by the online car booking driver are within a preset distance range.
Specifically, to avoid the driver uploading the vehicle appearance images in the database, rather than real-time vehicle appearance images, the present invention determines that the time difference between the acquisition time of the plurality of vehicle appearance images and the time of receiving the departure request is within a preset time period, such as 3 minutes, 5 minutes, etc., meanwhile, because the acquisition time of the plurality of vehicle appearance images is not shot at the same time, the earliest acquisition time in the acquisition time of the plurality of vehicle appearance images can be selected to be compared with the time of receiving the departure request, for example, the plurality of vehicle appearance images are collected at 8 am, 8 am 01 min, 8 am 02 min, 8 am 04 min, 8 am 06 min, the earliest acquisition time is 8 am 01 min, and 8 am 01 min is used to compare with the time when the departure request was received.
The driver is too close to the vehicle, which may cause the appearance pattern of the photographed angle to be incomplete; the method and the device have the advantages that the shooting position of the driver is far away from the vehicle, the occupation ratio of the vehicle in the picture is too small, the appearance pattern of the vehicle is fuzzy, and the cleanliness score is inaccurate, so that the shooting positions of the appearance images of the vehicles and the current position of the vehicle driven by the taxi driver are determined to be within the preset distance range, the problem that the occupation ratio of the vehicle in the appearance images is too small, the pattern of the vehicle is fuzzy can be avoided, meanwhile, the problems that the occupation ratio of the vehicle in the appearance images is too large, the pattern is incomplete are avoided, and the cleanliness accuracy is improved.
Further, the preset distance range may be 3-6 meters. Of course, the method can be customized according to the platform requirements, and the specific numerical value of the preset distance range is not specifically limited.
Based on the above description, one example is listed:
before a driver sends a car, the driver needs to shoot images of the front, the back, the left and the right of the car from car sending software of a mobile terminal and upload the images in real time, the car sending software can only shoot and upload the images immediately and cannot select the images from a picture library to upload the images, namely, the car sending software can judge whether the time difference between the acquisition time of a plurality of car appearance images and the time of receiving a car sending request is within a preset time length, if the images are not prompted to be not qualified, the images are uploaded normally within the preset time length;
when the images are uploaded, the departure software sends the current position information to the background server, and the current position information sent by the departure software to the background server is the current position of the vehicle driven by the online taxi appointment driver; the background server acquires the photographing position information from the uploaded image, compares the photographing position information with the current position of the vehicle driven by the online car booking driver, and judges that the uploaded picture is qualified when the position accords with a preset distance range;
then, the background server splices the multiple vehicle appearance images according to the shooting angle for shooting each vehicle appearance image to obtain a spliced image, segments the spliced image to obtain multiple segmented areas, inputs the segmented areas into a Transformer Encoder model, inputs a one-dimensional vector output by the Transformer Encoder model into an MLP model, judges whether the multiple uploaded images are the images of the same vehicle appearance shot by multiple shooting angles in the MLP model, and if so, detects the cleanliness of the vehicles in the images according to the multiple uploaded images, so that the situation that a driver only shoots clean angles and unclean angles cannot be completely presented can be avoided.
Based on the same inventive concept, as shown in fig. 5, an embodiment of the present invention provides a device for determining cleanliness of a vehicle exterior, including:
the segmentation module 500 is configured to segment the plurality of vehicle appearance images to obtain a plurality of segmented regions;
an extraction module 501, configured to perform feature extraction on each segmented region to obtain an image feature value corresponding to each segmented region, and determine a target image feature value according to the image feature value corresponding to each segmented region;
the detecting module 502 is configured to determine cleanliness of vehicles included in the plurality of vehicle appearance images according to the plurality of vehicle appearance images if it is determined that the plurality of vehicle appearance images are images of the same vehicle appearance captured at a plurality of capturing angles according to the target image feature value.
Optionally, the segmentation module 500 is specifically configured to:
and splicing the plurality of vehicle appearance images according to the shooting angle for shooting each vehicle appearance image to obtain a spliced image, and segmenting the spliced image to obtain a plurality of segmented areas.
Optionally, the apparatus further comprises:
the receiving module is used for receiving a departure request sent by a taxi appointment driver;
the state modification module is used for modifying the order receiving state of the vehicle driven by the vehicle ordering driver into a normal order receiving state if the cleanliness is larger than a preset cleanliness threshold; or
And if the cleanliness is not greater than the preset cleanliness threshold value, keeping the order receiving state of the vehicle driven by the network car booking driver unchanged, and informing the network car booking driver that the cleanliness of the vehicle driven by the network car booking driver is unqualified.
In an exemplary embodiment, there is also provided a storage medium, such as a memory, including instructions executable by a processor of a server to perform the vehicle appearance cleanliness determination method described above. Alternatively, the storage medium may be a non-transitory computer readable storage medium, which may be, for example, a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
An embodiment of the present invention further provides a server, including:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the vehicle appearance cleanliness determination method according to any one of the above.
In the embodiment of the present invention, besides the above-described structure, the structure of the server may also be as shown in fig. 6, where the server 600 includes: radio Frequency (RF) circuitry 610, a power supply 620, a processor 630, a memory 640, an input unit 650, a display unit 660, a communication interface 670, and a Wireless Fidelity (Wi-Fi) module 680. Those skilled in the art will appreciate that the configuration of the server shown in fig. 6 is not intended to be limiting, and that the servers provided in the embodiments of the present application may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components may be provided.
The following describes each component of the server 600 in detail with reference to fig. 6:
the RF circuit 610 may be used for receiving and transmitting data during a communication or conversation. In particular, the RF circuit 610 sends downlink data of a base station to the processor 630 for processing after receiving the downlink data; and in addition, sending the uplink data to be sent to the base station. Generally, the RF circuit 610 includes, but is not limited to, an antenna, at least one Amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like.
In addition, the RF circuit 610 may also communicate with networks and other terminals through wireless communication. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), and the like.
The Wi-Fi technology belongs to a short-distance wireless transmission technology, and the server 600 may connect to an Access Point (AP) through a Wi-Fi module 680, thereby implementing Access to a data network. The Wi-Fi module 680 may be used for receiving and transmitting data during communication.
The server 600 may be physically connected to other terminals through the communication interface 670. Optionally, the communication interface 670 is connected to the communication interface of the other terminal through a cable, so as to implement data transmission between the server 600 and the other terminal.
In the embodiment of the present application, the server 600 can implement a communication service to send information to other contacts, so the server 600 needs to have a data transmission function, that is, the server 600 needs to include a communication module inside. Although fig. 6 shows communication modules such as the RF circuit 610, the Wi-Fi module 680, and the communication interface 670, it is understood that at least one of the above components or other communication modules (such as a bluetooth module) for implementing communication exist in the server 600 for data transmission.
The memory 640 may be used to store software programs and modules. The processor 630 executes various functional applications and data processing of the server 600 by executing the software programs and modules stored in the memory 640, and when the processor 630 executes the program codes in the memory 640, part or all of the processes in fig. 1 or part or all of the processes in fig. 4 according to the embodiment of the present invention may be implemented.
Alternatively, the memory 640 may mainly include a program storage area and a data storage area. The storage program area can store an operating system, various application programs (such as communication application), a face recognition module and the like; the storage data area may store data (such as various multimedia files like pictures, video files, etc., and face information templates) created according to the use of the terminal, etc.
Further, the memory 640 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
The input unit 650 may be used to receive numeric or character information input by a user and generate key signal inputs related to user settings and function control of the server 600.
Alternatively, the input unit 650 may include a touch panel 651 and other input terminals 652.
The touch panel 651, also called a touch screen, may collect touch operations of a user (for example, operations of a user on or near the touch panel 651 by using any suitable object or accessory such as a finger or a stylus pen) and drive a corresponding connection device according to a preset program. Alternatively, the touch panel 651 may include two parts, a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 630, and can receive and execute commands sent by the processor 630. In addition, the touch panel 651 may be implemented in various types, such as resistive, capacitive, infrared, and surface acoustic wave.
Optionally, the other input terminals 652 may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
The display unit 660 may be used to display information input by a user or information provided to a user and various menus of the server 600. The display unit 660 is a display system of the server 600, and is used for presenting an interface to implement human-computer interaction.
The display unit 660 may include a display panel 661. Alternatively, the Display panel 661 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.
Further, the touch panel 651 can cover the display panel 661, and when the touch panel 651 detects a touch operation thereon or nearby, the touch operation is transmitted to the processor 630 to determine the type of touch event, and then the processor 630 provides a corresponding visual output on the display panel 661 according to the type of touch event.
Although in fig. 6, the touch panel 651 and the display panel 661 are provided as two separate components to implement the input and output functions of the server 600, in some embodiments, the touch panel 651 and the display panel 661 may be integrated to implement the input and output functions of the server 600.
The processor 630 is a control center of the server 600, connects each component using various interfaces and lines, performs various functions of the server 600 and processes data by running or executing software programs and/or modules stored in the memory 640 and calling data stored in the memory 640, thereby implementing various services based on the terminal.
Optionally, the processor 630 may include one or more processing units. Optionally, the processor 630 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, an application program, and the like, and the modem processor mainly processes wireless communication. It will be appreciated that the modem processor described above may not be integrated into the processor 630.
The server 600 also includes a power source 620 (such as a battery) for powering the various components. Optionally, the power supply 620 may be logically connected to the processor 630 through a power management system, so as to implement functions of managing charging, discharging, power consumption, and the like through the power management system.
The embodiment of the invention also provides a computer program product, and when the computer program product runs on a server, the server is enabled to execute the method for judging the cleanliness of the appearance of the vehicle, which is used for realizing any one of the methods in the embodiment of the invention.
Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.
Claims (10)
1. A method for determining vehicle appearance cleanliness, comprising:
segmenting a plurality of vehicle appearance images to obtain a plurality of segmented regions;
performing feature extraction on each segmented region to obtain an image feature value corresponding to each segmented region, and determining a target image feature value according to the image feature value corresponding to each segmented region;
if the plurality of vehicle appearance images are determined to be the images of the same vehicle appearance shot at a plurality of shooting angles according to the target image characteristic value, determining the cleanliness of the vehicles in the plurality of vehicle appearance images according to the plurality of vehicle appearance images;
determining the plurality of vehicle appearance images as images of the same vehicle appearance shot at a plurality of shooting angles according to the target image characteristic values in the following way:
if the target image characteristic values representing the environmental characteristics in the target image characteristic values are the same, and the target image characteristic values representing the vehicle body characteristics of the same type in the target image characteristic values are the same, determining that the plurality of vehicle appearance images are the images of the same vehicle appearance shot at a plurality of shooting angles; wherein the types of body features include some or all of the following: wheel, car light, automobile body, handlebar.
2. The vehicle appearance cleanliness determination method according to claim 1, wherein the step of dividing the plurality of vehicle appearance images to obtain the plurality of divided regions comprises:
and splicing the plurality of vehicle appearance images according to the shooting angle of each vehicle appearance image to obtain a spliced image, and segmenting the spliced image to obtain a plurality of segmented areas.
3. The vehicle appearance cleanliness determination method according to claim 1 or 2, wherein before the plurality of vehicle appearance images are divided into a plurality of divided regions, the method further comprises:
receiving a departure request sent by a network car booking driver;
after determining the cleanliness of the vehicles included in the plurality of vehicle appearance images from the plurality of vehicle appearance images, the method further includes:
if the cleanliness is larger than a preset cleanliness threshold value, the order receiving state of the vehicle driven by the taxi appointment driver is changed into a normal order receiving state; or
And if the cleanliness is not greater than the preset cleanliness threshold value, keeping the order receiving state of the vehicle driven by the network car booking driver unchanged, and informing the network car booking driver that the cleanliness of the vehicle driven by the network car booking driver is unqualified.
4. The vehicle appearance cleanliness determination method according to claim 3, wherein before the plurality of vehicle appearance images are divided into a plurality of divided regions, the method further comprises:
determining that the time difference between the acquisition time of the plurality of vehicle appearance images and the time of receiving the departure request is within a preset time length; and/or
And determining that the shooting positions of the plurality of vehicle appearance images and the current position of the vehicle driven by the online car booking driver are within a preset distance range.
5. A vehicle appearance cleanliness determination device characterized by comprising:
the segmentation module is used for segmenting the plurality of vehicle appearance images to obtain a plurality of segmented areas;
the extraction module is used for extracting features of each segmented region to obtain an image feature value corresponding to each segmented region, and determining a target image feature value according to the image feature value corresponding to each segmented region;
the detection module is used for determining the cleanliness of the vehicles in the multiple vehicle appearance images according to the multiple vehicle appearance images if the multiple vehicle appearance images are determined to be the images of the same vehicle appearance shot at multiple shooting angles according to the target image characteristic value;
determining the plurality of vehicle appearance images as images of the same vehicle appearance shot at a plurality of shooting angles according to the target image characteristic values in the following way:
if the target image characteristic values representing the environmental characteristics in the target image characteristic values are the same, and the target image characteristic values representing the vehicle body characteristics of the same type in the target image characteristic values are the same, determining that the plurality of vehicle appearance images are the images of the same vehicle appearance shot at a plurality of shooting angles; wherein the types of body features include some or all of the following: wheel, car light, automobile body, handlebar.
6. The vehicle appearance cleanliness determination apparatus according to claim 5, wherein the division module is specifically configured to:
and splicing the plurality of vehicle appearance images according to the shooting angle of each vehicle appearance image to obtain a spliced image, and segmenting the spliced image to obtain a plurality of segmented areas.
7. The vehicle appearance cleanliness determination device according to claim 5 or 6, characterized by further comprising:
the receiving module is used for receiving a departure request sent by a network car booking driver;
the state modification module is used for modifying the order receiving state of the vehicle driven by the vehicle ordering driver into a normal order receiving state if the cleanliness is larger than a preset cleanliness threshold; or
And if the cleanliness is not greater than the preset cleanliness threshold value, keeping the order receiving state of the vehicle driven by the network car booking driver unchanged, and informing the network car booking driver that the cleanliness of the vehicle driven by the network car booking driver is unqualified.
8. The vehicle exterior cleanliness determination device according to claim 7, further comprising:
the determining module is used for determining that the time difference between the acquisition time of the plurality of vehicle appearance images and the time of receiving the departure request is within a preset time length; and/or determining that the shooting positions of the plurality of vehicle appearance images and the current position of the vehicle driven by the net appointment driver are within a preset distance range.
9. A server, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the vehicle appearance cleanliness determination method according to any one of claim 1 to claim 4.
10. A storage medium, comprising:
the instructions in the storage medium, when executed by a processor of a server, enable the processor to perform the vehicle appearance cleanliness determination method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110524024.7A CN113408345B (en) | 2021-05-13 | 2021-05-13 | Vehicle appearance cleanliness judgment method and device, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110524024.7A CN113408345B (en) | 2021-05-13 | 2021-05-13 | Vehicle appearance cleanliness judgment method and device, server and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113408345A CN113408345A (en) | 2021-09-17 |
| CN113408345B true CN113408345B (en) | 2022-06-10 |
Family
ID=77678476
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110524024.7A Active CN113408345B (en) | 2021-05-13 | 2021-05-13 | Vehicle appearance cleanliness judgment method and device, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113408345B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110276258A (en) * | 2019-05-21 | 2019-09-24 | 智慧互通科技有限公司 | A kind of method and system identifying vehicle appearance |
| CN111914664A (en) * | 2020-07-06 | 2020-11-10 | 同济大学 | Vehicle multi-target detection and track tracking method based on re-identification |
| CN112381477A (en) * | 2021-01-18 | 2021-02-19 | 南京领行科技股份有限公司 | Vehicle appearance cleanliness monitoring method and device, server and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109145903A (en) * | 2018-08-22 | 2019-01-04 | 阿里巴巴集团控股有限公司 | A kind of image processing method and device |
-
2021
- 2021-05-13 CN CN202110524024.7A patent/CN113408345B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110276258A (en) * | 2019-05-21 | 2019-09-24 | 智慧互通科技有限公司 | A kind of method and system identifying vehicle appearance |
| CN111914664A (en) * | 2020-07-06 | 2020-11-10 | 同济大学 | Vehicle multi-target detection and track tracking method based on re-identification |
| CN112381477A (en) * | 2021-01-18 | 2021-02-19 | 南京领行科技股份有限公司 | Vehicle appearance cleanliness monitoring method and device, server and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113408345A (en) | 2021-09-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3471021A1 (en) | Method for determining target intelligently followed by unmanned aerial vehicle, unmanned aerial vehicle and remote controller | |
| CN110471058A (en) | The system and method detected automatically for trailer attribute | |
| CN106407984B (en) | Target object identification method and device | |
| CN109977845B (en) | Driving region detection method and vehicle-mounted terminal | |
| US11204604B2 (en) | Remote driving managing apparatus and computer-readable storage medium | |
| CN111837158A (en) | Image processing method and device, shooting device and movable platform | |
| CN113688805B (en) | Unmanned aerial vehicle-based unlicensed muck vehicle identification method and system | |
| CN112381477B (en) | Vehicle appearance cleanliness monitoring method and device, server and storage medium | |
| CN112613348A (en) | Character recognition method and electronic equipment | |
| CN111931683B (en) | Image recognition method, device and computer readable storage medium | |
| CN111127902A (en) | Parking lot vehicle positioning system and method based on image recognition | |
| CN111627057A (en) | Distance measuring method and device and server | |
| CN113793413A (en) | Three-dimensional reconstruction method and device, electronic equipment and storage medium | |
| CN114359789A (en) | Target detection method, device, equipment and medium for video image | |
| CN113408345B (en) | Vehicle appearance cleanliness judgment method and device, server and storage medium | |
| CN108230680B (en) | Vehicle behavior information acquisition method and device and terminal | |
| CN110660000A (en) | Data prediction method, device, equipment and computer readable storage medium | |
| CN112069901A (en) | In-vehicle article monitoring method, electronic device, and storage medium | |
| CN111898475A (en) | Method and device for estimating state of non-motor vehicle, storage medium, and electronic device | |
| CN111723614A (en) | Traffic signal lamp identification method and device | |
| CN114373081A (en) | Image processing method and device, electronic device and storage medium | |
| CN114267021A (en) | Object recognition method and device, storage medium and electronic equipment | |
| CN110263721B (en) | Vehicle lamp setting method and device | |
| JP2021152826A (en) | Information processing device, subject classification method, and subject classification program | |
| CN113449629A (en) | Lane line false and true identification device, method, equipment and medium based on driving video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |