CN115147838B - Image processing method, device, vehicle, medium, and program product - Google Patents
Image processing method, device, vehicle, medium, and program product Download PDFInfo
- Publication number
- CN115147838B CN115147838B CN202210772940.7A CN202210772940A CN115147838B CN 115147838 B CN115147838 B CN 115147838B CN 202210772940 A CN202210772940 A CN 202210772940A CN 115147838 B CN115147838 B CN 115147838B
- Authority
- CN
- China
- Prior art keywords
- target
- image
- images
- obstacle
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 16
- 230000007613 environmental effect Effects 0.000 claims abstract description 159
- 238000013145 classification model Methods 0.000 claims abstract description 47
- 238000012545 processing Methods 0.000 claims abstract description 11
- 238000000034 method Methods 0.000 claims description 53
- 238000004590 computer program Methods 0.000 claims description 24
- 238000001514 detection method Methods 0.000 claims description 20
- 230000008569 process Effects 0.000 claims description 19
- 238000012549 training Methods 0.000 claims description 11
- 238000002372 labelling Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 abstract description 9
- 238000004891 communication Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 8
- 230000003068 static effect Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 230000009471 action Effects 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Databases & Information Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
Abstract
The present disclosure relates to the field of automatic driving technology, and provides an image processing method, an image processing device, a vehicle, a medium and a program product. In some embodiments of the present disclosure, a plurality of environmental images during automatic driving of a vehicle are acquired; detecting the obstacle of the plurality of environmental images to obtain target obstacles contained in the plurality of environmental images and positions of the target obstacles, wherein the target obstacles exist in the plurality of environmental images together; splicing a plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain a target spliced image; inputting the target spliced image into the trained object classification model to obtain a classification result of the target obstacle, classifying the obstacle by using the target spliced image and the object classification model, and improving classification accuracy of the obstacle.
Description
Technical Field
The present disclosure relates to the field of autopilot technology, and in particular, to an image processing method, apparatus, vehicle, medium, and program product.
Background
With the rapid development of the automobile industry in China and the improvement of the living standard of people, the automobile possession of resident families is rapidly increased, and the automobile gradually becomes one of the indispensable transportation means in the life of people.
During automatic driving of a vehicle, obstacles in the surrounding environment fall into two categories: dynamic obstacles and static obstacles. Dynamic obstacles, such as other vehicles, non-motor vehicles, pedestrians, etc. that travel. Static obstacles, such as brake bars, other vehicles in a stationary state.
Currently, vehicles have low accuracy in classifying obstacles in the surrounding environment during automatic driving.
Disclosure of Invention
The present disclosure provides an image processing method, apparatus, vehicle, medium and program product to at least solve the problem in the related art that the classification accuracy of the vehicle to the obstacle in the surrounding environment is low. The technical scheme of the present disclosure is as follows:
the embodiment of the disclosure provides an image processing method, which comprises the following steps:
acquiring a plurality of environment images in the automatic driving process of a vehicle;
detecting the obstacle of the plurality of environmental images to obtain target obstacles contained in the plurality of environmental images and positions of the target obstacles, wherein the target obstacles exist in the plurality of environmental images together;
splicing the plurality of environment images according to the target obstacle and the position of the target obstacle to obtain a target spliced image;
and inputting the target spliced image into a trained object classification model to obtain a classification result of the target obstacle.
Optionally, the detecting the obstacle for the plurality of environmental images to obtain a target obstacle included in the plurality of environmental images and a position of the target obstacle includes:
respectively detecting obstacles in each environmental image in the plurality of environmental images to obtain the obstacles and the positions of the obstacles contained in each environmental image;
and selecting a target obstacle and a position of the target obstacle which are commonly existing in each environment image from all the obstacles contained in the environment images.
Optionally, detecting an obstacle in each of the plurality of environmental images to obtain an obstacle and an obstacle position contained in each environmental image, including:
and inputting each environment image into the trained obstacle detection model to obtain the obstacle and the obstacle position contained in each environment image.
Optionally, the stitching the plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain a target stitched image includes:
aiming at a target image, splicing the target obstacle contained in other environment images on the target image according to the positions of the target obstacle contained in other environment images to obtain the target spliced image;
the target image is any one of the plurality of environment images, and the other environment images are environment images except the target image.
Optionally, before using the object classification model, the method further comprises:
collecting a plurality of historical sample environment images of a sample vehicle in the running process;
grouping the plurality of historical sample environment images to obtain a plurality of environment image groups, wherein each environment image group comprises a plurality of images;
splicing each environmental image group to obtain a plurality of sample spliced images;
performing obstacle labeling on the plurality of sample spliced images to obtain sample obstacles in the plurality of sample spliced images, the positions of the sample obstacles and the types of the sample obstacles;
and training an initial model according to the plurality of sample spliced images, sample obstacles in the plurality of sample spliced images, the positions of the sample obstacles and the types of the sample obstacles to obtain the object classification model.
Optionally, the stitching is performed on each environmental image group to obtain a plurality of sample stitched images, including:
for a target sample image, according to the position of the vehicle in each environment image group, splicing the other sample environment images with the target sample image to obtain a plurality of sample spliced images;
the target sample image is any one sample environmental image in each environmental image group, and the other sample environmental images are sample environmental images except the target sample image in each environmental image group.
Optionally, after the inputting the target stitched image into the object classification model that has been trained, obtaining a classification result of the target obstacle, the method further includes:
and controlling the vehicle to automatically drive according to the classification result of the target obstacle.
The embodiment of the disclosure also provides an image processing apparatus, including:
the acquisition module is used for acquiring a plurality of environment images in the automatic driving process of the vehicle;
the detection module is used for detecting the obstacles of the plurality of environmental images to obtain target obstacles contained in the plurality of environmental images and positions of the target obstacles, wherein the target obstacles coexist in the plurality of environmental images;
the splicing module is used for splicing the plurality of environment images according to the target obstacle and the position of the target obstacle to obtain a target spliced image;
and the classification model module is used for inputting the target spliced image into a trained object classification model to obtain a classification result of the target obstacle.
The disclosed embodiments also provide a vehicle including: a memory and a processor;
the memory is used for storing a computer program;
the processor is configured to execute the computer program to implement the steps in the method as described above.
The disclosed embodiments also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the above-described method.
The disclosed embodiments also provide a computer program product comprising a computer program/instructions, characterized in that the computer program/instructions, when executed by a processor, implement the steps of the above-described method.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:
in some embodiments of the present disclosure, a plurality of environmental images during automatic driving of a vehicle are acquired; detecting the obstacle of the plurality of environmental images to obtain target obstacles contained in the plurality of environmental images and positions of the target obstacles, wherein the target obstacles exist in the plurality of environmental images together; splicing a plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain a target spliced image; inputting the target spliced image into the trained object classification model to obtain a classification result of the target obstacle, classifying the obstacle by using the target spliced image and the object classification model, and improving classification accuracy of the obstacle.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure and do not constitute an undue limitation on the disclosure.
Fig. 1 is a flowchart of an image processing method according to an exemplary embodiment of the present disclosure;
FIG. 2 is a flow chart of another image processing method according to an exemplary embodiment of the present disclosure;
FIG. 3 is a flow chart of another image processing method according to an exemplary embodiment of the present disclosure;
fig. 4 is a schematic structural view of an image processing apparatus according to an exemplary embodiment of the present disclosure;
fig. 5 is a schematic structural view of a vehicle according to an exemplary embodiment of the present disclosure.
Detailed Description
In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
It should be noted that, the user information related to the present disclosure includes, but is not limited to: user equipment information and user personal information; the processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the user information in the present disclosure all conform to the regulations of the relevant laws and regulations and do not violate the well-known and popular public order.
With the rapid development of the automobile industry in China and the improvement of the living standard of people, the automobile possession of resident families is rapidly increased, and the automobile gradually becomes one of the indispensable transportation means in the life of people.
During automatic driving of a vehicle, obstacles in the surrounding environment fall into two categories: dynamic obstacles and static obstacles. Dynamic obstacles, such as other vehicles, non-motor vehicles, pedestrians, etc. that travel. Static obstacles, such as brake bars, other vehicles in a stationary state.
Currently, vehicles have low accuracy in classifying obstacles in the surrounding environment during automatic driving.
In view of the above-mentioned technical problems, in some embodiments of the present disclosure, a plurality of environmental images during automatic driving of a vehicle are acquired; detecting the obstacle of the plurality of environmental images to obtain target obstacles contained in the plurality of environmental images and positions of the target obstacles, wherein the target obstacles exist in the plurality of environmental images together; splicing a plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain a target spliced image; inputting the target spliced image into the trained object classification model to obtain a classification result of the target obstacle, classifying the obstacle by using the target spliced image and the object classification model, and improving classification accuracy of the obstacle.
The following describes in detail the technical solutions provided by the embodiments of the present disclosure with reference to the accompanying drawings.
Fig. 1 is a flowchart of an image processing method according to an exemplary embodiment of the present disclosure. As shown in fig. 1, the image processing method includes:
s101: acquiring a plurality of environment images in the automatic driving process of a vehicle;
s102: detecting the obstacle of the plurality of environmental images to obtain target obstacles contained in the plurality of environmental images and positions of the target obstacles, wherein the target obstacles exist in the plurality of environmental images together;
s103: splicing a plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain a target spliced image;
s104: and inputting the target spliced image into the trained object classification model to obtain a classification result of the target obstacle.
In this embodiment, the execution subject of the above method is a vehicle. The embodiments of the present disclosure are not limited to the type of vehicle, including but not limited to any of the following: electric vehicles, oil-powered vehicles, hybrid electric vehicles, hydrogen-powered vehicles, hybrid electric vehicles with new energy and oil-powered, and the like.
In this embodiment, the vehicle acquires a plurality of environmental images during its automatic driving; detecting the obstacle of the plurality of environmental images to obtain target obstacles contained in the plurality of environmental images and positions of the target obstacles, wherein the target obstacles exist in the plurality of environmental images together; splicing a plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain a target spliced image; inputting the target spliced image into the trained object classification model to obtain a classification result of the target obstacle, classifying the obstacle by using the target spliced image and the object classification model, and improving classification accuracy of the obstacle.
In the above-described embodiment, the vehicle collects an environmental image in front of the vehicle using the camera mounted in front of the vehicle. The embodiment of the disclosure does not limit the number of the plurality of environment images, and can be adjusted according to actual conditions. A plurality of environmental images, for example, 3 environmental images, 5 environmental images, 10 environmental images, and the like. The plurality of environmental images may be a plurality of consecutive frames of environmental images acquired.
In the above embodiment, the obstacle detection is performed on the plurality of environmental images, and the target obstacle and the position of the target obstacle included in the plurality of environmental images are obtained. One implementation method is that obstacle detection is carried out on each environmental image in a plurality of environmental images respectively to obtain an obstacle and an obstacle position contained in each environmental image; the target obstacle and the position of the target obstacle which coexist in each environmental image are selected from all the obstacles contained in the plurality of environmental images.
For example, obstacle detection is performed on consecutive 3-frame environmental images, and an obstacle position included in the 3-frame environmental images are obtained, respectively. The 1 st frame of environment image comprises a gate, a first vehicle and a second vehicle; the 2 nd frame environment image comprises a gate and a first vehicle; the 3 rd frame of environment image includes a gate, a first vehicle and a second vehicle. Selecting a target obstacle which is coexistent in the 3-frame environment image from the 3-frame environment image: the method comprises the steps of acquiring the position of a gate and the position of a first vehicle.
Optionally, detecting the obstacle of each environmental image in the plurality of environmental images respectively to obtain the obstacle and the obstacle position contained in each environmental image. One way to achieve this is to input each environmental image into a trained obstacle detection model to obtain the obstacle and the obstacle position contained in each environmental image.
In the above embodiment, the plurality of environmental images are stitched according to the target obstacle and the position of the target obstacle, so as to obtain the target stitched image. One implementation way is that, for a target image, target obstacles contained in other environment images are spliced on the target image according to the positions of the target obstacles contained in other environment images, so as to obtain a target spliced image; the target image is any one of the plurality of environment images, and the other environment images are environment images except the target image.
For example, for consecutive 3-frame environmental images, the 1 st-frame environmental image is taken as a target image; and splicing the gate and the first vehicle contained in the other two frames of environment images on the target image according to the positions of the gate and the first vehicle contained in the other two frames of environment images, so as to obtain the target spliced image.
In the above embodiment, the target stitched image is input into the object classification model that has been trained, and the classification result of the target obstacle is obtained. The classification result of the target obstacle includes a dynamic obstacle and a static obstacle.
For example, the target spliced image is input into a trained object classification model to obtain a classification result that the gate is a static obstacle and the first vehicle is a dynamic obstacle.
Before using the object classification model, a description is given of a training process of the object classification model:
manufacturing a test sample set and a training sample set;
firstly, collecting a plurality of historical sample environment images of a large number of sample vehicles in the running process, and collecting the historical sample environment images of various types of vehicles and various environments as much as possible so as to improve the sample coverage rate. Grouping a plurality of historical sample environment images to obtain a plurality of environment image groups, wherein each environment image group comprises a plurality of images; and splicing each environmental image group to obtain a plurality of sample spliced images. Then, carrying out obstacle marking on part of the sample spliced images to obtain sample obstacles in the plurality of sample spliced images, the positions of the sample obstacles, the types of the sample obstacles and the plurality of sample spliced images, and taking the sample obstacles, the positions of the sample obstacles, the types of the sample obstacles and the plurality of sample spliced images as a training sample set; and the other part of unlabeled sample is spliced into an image to be used as a test sample set.
The method comprises the steps of aiming at a target sample image, splicing other sample environment images with the target sample image according to the positions of vehicles in each environment image group to obtain a plurality of sample spliced images; the target sample image is any one sample environmental image in each environmental image group, and the other sample environmental images are sample environmental images except the target sample image in each environmental image group.
And (5) building an initial model.
Suitable classification models, such as logistic regression models, decision tree models, support vector machine models, and naive bayes models, are selected.
And training the network parameters of the initial model by using the training sample set to obtain an object classification model.
And inputting the plurality of sample spliced images and the sample obstacles in the plurality of sample spliced images, the positions of the sample obstacles and the types of the sample obstacles into an initial model. And in the initial model, performing operations such as calculation on classification of the sample obstacle according to network parameters of the initial model, and outputting a classification result of the sample obstacle by an output layer of the initial model. And the loss function layer of the initial model can calculate a loss function according to the difference between the classification result output by the output layer and the real type of the sample obstacle. If the loss function does not meet the set requirement, the model parameters can be adjusted, and iterative training is continued. When the loss function of the initial model meets the set requirement, the object classification model after training is obtained.
In the above embodiment, the vehicle is controlled to automatically drive after the classification result of the target obstacle is obtained.
In connection with the above description of the embodiments, fig. 2 is a schematic flow chart of another image processing method according to an exemplary embodiment of the disclosure. As shown in fig. 2, the image processing method includes:
s201: acquiring a plurality of environment images in the automatic driving process of a vehicle;
s202: respectively detecting obstacles in each environmental image in the plurality of environmental images to obtain the obstacles and the positions of the obstacles contained in each environmental image;
s203: selecting a target obstacle and the position of the target obstacle which coexist in each environmental image from all the obstacles contained in the environmental images;
s204: splicing a plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain a target spliced image;
s205: and inputting the target spliced image into the trained object classification model to obtain a classification result of the target obstacle.
In this embodiment, the execution subject of the above method is a vehicle. The embodiments of the present disclosure are not limited to the type of vehicle, including but not limited to any of the following: electric vehicles, oil-powered vehicles, hybrid electric vehicles, hydrogen-powered vehicles, hybrid electric vehicles with new energy and oil-powered, and the like.
In this embodiment, each step of the method may refer to the description of the corresponding part in each embodiment, and meanwhile, the corresponding beneficial technical effects may be obtained, which is not described herein.
In conjunction with the above descriptions of the embodiments, fig. 3 is a schematic flow chart of another image processing method according to an exemplary embodiment of the disclosure. As shown in fig. 3, the image processing method includes:
s301: acquiring a plurality of environment images in the automatic driving process of a vehicle;
s302: detecting the obstacle of the plurality of environmental images to obtain target obstacles contained in the plurality of environmental images and positions of the target obstacles, wherein the target obstacles exist in the plurality of environmental images together;
s303: splicing a plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain a target spliced image;
s304: inputting the target spliced image into a trained object classification model to obtain a classification result of a target obstacle;
s305: and controlling the automatic driving of the vehicle according to the classification result of the target obstacle.
In this embodiment, the execution subject of the above method is a vehicle. The embodiments of the present disclosure are not limited to the type of vehicle, including but not limited to any of the following: electric vehicles, oil-powered vehicles, hybrid electric vehicles, hydrogen-powered vehicles, hybrid electric vehicles with new energy and oil-powered, and the like.
In this embodiment, each step of the method may refer to the description of the corresponding part in each embodiment, and meanwhile, the corresponding beneficial technical effects may be obtained, which is not described herein.
In the above-described method embodiments of the present disclosure, a plurality of environmental images during automatic driving of a vehicle are acquired; detecting the obstacle of the plurality of environmental images to obtain target obstacles contained in the plurality of environmental images and positions of the target obstacles, wherein the target obstacles exist in the plurality of environmental images together; splicing a plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain a target spliced image; inputting the target spliced image into the trained object classification model to obtain a classification result of the target obstacle, classifying the obstacle by using the target spliced image and the object classification model, and improving classification accuracy of the obstacle.
Fig. 4 is a schematic structural view of an image processing apparatus 40 according to an exemplary embodiment of the present disclosure. As shown in fig. 4, the image processing apparatus 40 includes: the system comprises an acquisition module 41, a detection module 42, a splicing module 43 and a classification model module 44.
Wherein, the acquisition module 41 is configured to acquire a plurality of environmental images during automatic driving of the vehicle;
the detection module 42 is configured to perform obstacle detection on a plurality of environmental images, so as to obtain a target obstacle and a position of the target obstacle included in the plurality of environmental images, where the target obstacle exists in the plurality of environmental images together;
a stitching module 43, configured to stitch a plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain a target stitched image;
the classification model module 44 is configured to input the target stitched image into a trained object classification model, so as to obtain a classification result of the target obstacle.
Optionally, the detection module 42 is configured to, when performing obstacle detection on the plurality of environmental images to obtain a target obstacle and a position of the target obstacle included in the plurality of environmental images:
respectively detecting obstacles in each environmental image in the plurality of environmental images to obtain the obstacles and the positions of the obstacles contained in each environmental image;
the target obstacle and the position of the target obstacle which coexist in each environmental image are selected from all the obstacles contained in the plurality of environmental images.
Optionally, the detection module 42 is configured to, when performing obstacle detection on each of the plurality of environmental images to obtain an obstacle and an obstacle position included in each environmental image, respectively:
and inputting each environmental image into the trained obstacle detection model to obtain the obstacle and the obstacle position contained in each environmental image.
Optionally, the stitching module 43 is configured to, when stitching the plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain the target stitched image:
aiming at the target image, splicing the target obstacle contained in other environment images on the target image according to the positions of the target obstacle contained in other environment images to obtain a target spliced image;
the target image is any one of the plurality of environment images, and the other environment images are environment images except the target image.
Optionally, before using the object classification model, classification model module 44 may be further configured to:
collecting a plurality of historical sample environment images of a sample vehicle in the running process;
grouping a plurality of historical sample environment images to obtain a plurality of environment image groups, wherein each environment image group comprises a plurality of images;
each environmental image group is spliced to obtain a plurality of sample spliced images;
performing obstacle labeling on the plurality of sample spliced images to obtain sample obstacles in the plurality of sample spliced images, positions of the sample obstacles and types of the sample obstacles;
and training the initial model according to the plurality of sample spliced images and the sample obstacles in the plurality of sample spliced images, the positions of the sample obstacles and the types of the sample obstacles to obtain an object classification model.
Optionally, the classification model module 44 is configured to, when stitching each environmental image group to obtain a plurality of sample stitched images:
aiming at the target sample images, according to the positions of vehicles in each environment image group, splicing other sample environment images with the target sample images to obtain a plurality of sample spliced images;
the target sample image is any one sample environmental image in each environmental image group, and the other sample environmental images are sample environmental images except the target sample image in each environmental image group.
Optionally, the classification model module 44 may be further configured to, after inputting the target stitched image into the object classification model that has been trained, obtain a classification result of the target obstacle:
and controlling the automatic driving of the vehicle according to the classification result of the target obstacle.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
Fig. 5 is a schematic structural view of a vehicle 50 provided in an exemplary embodiment of the present disclosure. As shown in fig. 5, the vehicle 50 includes: a memory 51 and a processor 52. In addition, the vehicle further includes a power supply assembly 53 and a communication assembly 54.
The memory 51 is used for storing a computer program and may be configured to store other various data to support operations on the user terminal. Examples of such data include instructions for any application or method operating on a user terminal.
The memory 51 may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
A communication component 54 for data transmission with other devices.
A processor 52, executable computer instructions stored in memory 51, for: acquiring a plurality of environment images in the automatic driving process of a vehicle;
detecting the obstacle of the plurality of environmental images to obtain target obstacles contained in the plurality of environmental images and positions of the target obstacles, wherein the target obstacles exist in the plurality of environmental images together;
splicing a plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain a target spliced image;
and inputting the target spliced image into the trained object classification model to obtain a classification result of the target obstacle.
Optionally, when performing obstacle detection on the plurality of environmental images to obtain the target obstacle and the positions of the target obstacle included in the plurality of environmental images, the processor 52 is configured to:
respectively detecting obstacles in each environmental image in the plurality of environmental images to obtain the obstacles and the positions of the obstacles contained in each environmental image;
the target obstacle and the position of the target obstacle which coexist in each environmental image are selected from all the obstacles contained in the plurality of environmental images.
Optionally, when performing obstacle detection on each of the plurality of environmental images to obtain an obstacle and an obstacle position included in each environmental image, the processor 52 is configured to:
and inputting each environmental image into the trained obstacle detection model to obtain the obstacle and the obstacle position contained in each environmental image.
Optionally, the processor 52 is configured to, when stitching the plurality of environmental images according to the target obstacle and the position of the target obstacle, obtain a target stitched image:
aiming at the target image, splicing the target obstacle contained in other environment images on the target image according to the positions of the target obstacle contained in other environment images to obtain a target spliced image;
the target image is any one of the plurality of environment images, and the other environment images are environment images except the target image.
Optionally, before using the object classification model, the processor 52 may be further configured to:
collecting a plurality of historical sample environment images of a sample vehicle in the running process;
grouping a plurality of historical sample environment images to obtain a plurality of environment image groups, wherein each environment image group comprises a plurality of images;
each environmental image group is spliced to obtain a plurality of sample spliced images;
performing obstacle labeling on the plurality of sample spliced images to obtain sample obstacles in the plurality of sample spliced images, positions of the sample obstacles and types of the sample obstacles;
and training the initial model according to the plurality of sample spliced images and the sample obstacles in the plurality of sample spliced images, the positions of the sample obstacles and the types of the sample obstacles to obtain an object classification model.
Optionally, the processor 52, when stitching each ambient image group to obtain a plurality of sample stitched images, is configured to:
aiming at the target sample images, according to the positions of vehicles in each environment image group, splicing other sample environment images with the target sample images to obtain a plurality of sample spliced images;
the target sample image is any one sample environmental image in each environmental image group, and the other sample environmental images are sample environmental images except the target sample image in each environmental image group.
Optionally, after inputting the target stitched image into the object classification model that has been trained, the processor 52 may be further configured to:
and controlling the automatic driving of the vehicle according to the classification result of the target obstacle.
Accordingly, the disclosed embodiments also provide a computer-readable storage medium storing a computer program. The computer-readable storage medium stores a computer program that, when executed by one or more processors, causes the one or more processors to perform the steps in the method embodiment of fig. 1.
Accordingly, the disclosed embodiments also provide a computer program product comprising a computer program/instructions for executing the steps of the method embodiment of fig. 1 by a processor.
The communication assembly of fig. 5 is configured to facilitate wired or wireless communication between the device in which the communication assembly is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile communication network of WiFi,2G, 3G, 4G/LTE, 5G, etc., or a combination thereof. In one exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
The power supply assembly shown in fig. 5 provides power for various components of the device in which the power supply assembly is located. The power components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the devices in which the power components are located.
The vehicle may further include a display screen and an audio assembly.
The display screen includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or sliding action, but also the duration and pressure associated with the touch or sliding operation.
An audio component, which may be configured to output and/or input an audio signal. For example, the audio component includes a Microphone (MIC) configured to receive external audio signals when the device in which the audio component is located is in an operational mode, such as a call mode, a recording mode, and a speech recognition mode. The received audio signal may be further stored in a memory or transmitted via a communication component. In some embodiments, the audio assembly further comprises a speaker for outputting audio signals.
In the above-described apparatus, vehicle, storage medium, and program product embodiments of the present disclosure, a plurality of environmental images during automatic driving of a vehicle are acquired; detecting the obstacle of the plurality of environmental images to obtain target obstacles contained in the plurality of environmental images and positions of the target obstacles, wherein the target obstacles exist in the plurality of environmental images together; splicing a plurality of environmental images according to the target obstacle and the position of the target obstacle to obtain a target spliced image; inputting the target spliced image into the trained object classification model to obtain a classification result of the target obstacle, classifying the obstacle by using the target spliced image and the object classification model, and improving classification accuracy of the obstacle.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.
The above is merely a specific embodiment of the disclosure to enable one skilled in the art to understand or practice the disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (9)
1. An image processing method, comprising:
acquiring a plurality of environment images in the automatic driving process of a vehicle;
respectively detecting obstacles in each environmental image in the plurality of environmental images to obtain the obstacles and the positions of the obstacles contained in each environmental image;
selecting a target obstacle and a position of the target obstacle which are commonly present in each environmental image from all the obstacles contained in the environmental images;
aiming at a target image, splicing the target obstacle contained in other environment images on the target image according to the positions of the target obstacle contained in other environment images to obtain a target spliced image;
the target image is any one of the plurality of environment images, and the other environment images are environment images except the target image;
and inputting the target spliced image into a trained object classification model to obtain a classification result of the target obstacle.
2. The method according to claim 1, wherein performing obstacle detection on each of the plurality of environmental images to obtain an obstacle and an obstacle position included in each environmental image includes:
and inputting each environment image into the trained obstacle detection model to obtain the obstacle and the obstacle position contained in each environment image.
3. The method of claim 1, wherein prior to using the object classification model, the method further comprises:
collecting a plurality of historical sample environment images of a sample vehicle in the running process;
grouping the plurality of historical sample environment images to obtain a plurality of environment image groups, wherein each environment image group comprises a plurality of images;
splicing each environmental image group to obtain a plurality of sample spliced images;
performing obstacle labeling on the plurality of sample spliced images to obtain sample obstacles in the plurality of sample spliced images, the positions of the sample obstacles and the types of the sample obstacles;
and training an initial model according to the plurality of sample spliced images, sample obstacles in the plurality of sample spliced images, the positions of the sample obstacles and the types of the sample obstacles to obtain the object classification model.
4. A method according to claim 3, wherein said stitching each of said ambient image groups to obtain a plurality of sample stitched images comprises:
for a target sample image, according to the position of the vehicle in each environment image group, splicing the other sample environment images with the target sample image to obtain a plurality of sample spliced images;
the target sample image is any one sample environmental image in each environmental image group, and the other sample environmental images are sample environmental images except the target sample image in each environmental image group.
5. The method of claim 1, wherein after the inputting the target stitched image into the trained object classification model to obtain the classification result of the target obstacle, the method further comprises:
and controlling the vehicle to automatically drive according to the classification result of the target obstacle.
6. An image processing apparatus, comprising:
the acquisition module is used for acquiring a plurality of environment images in the automatic driving process of the vehicle;
the detection module is used for detecting the obstacle of each environmental image in the plurality of environmental images respectively to obtain the obstacle and the obstacle position contained in each environmental image; selecting a target obstacle and a position of the target obstacle which are commonly present in each environmental image from all the obstacles contained in the environmental images;
the splicing module is used for splicing the target obstacles contained in the other environment images on the target image according to the positions of the target obstacles contained in the other environment images to obtain a target spliced image; the target image is any one of the plurality of environment images, and the other environment images are environment images except the target image;
and the classification model module is used for inputting the target spliced image into a trained object classification model to obtain a classification result of the target obstacle.
7. A vehicle, characterized by comprising: a memory and a processor;
the memory is used for storing a computer program;
the processor for executing the computer program to carry out the steps of the method according to any one of claims 1 to 5.
8. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1-5.
9. A computer program product comprising computer programs/instructions which, when executed by a processor, implement the steps of the method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210772940.7A CN115147838B (en) | 2022-06-30 | 2022-06-30 | Image processing method, device, vehicle, medium, and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210772940.7A CN115147838B (en) | 2022-06-30 | 2022-06-30 | Image processing method, device, vehicle, medium, and program product |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115147838A CN115147838A (en) | 2022-10-04 |
| CN115147838B true CN115147838B (en) | 2023-08-29 |
Family
ID=83410009
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210772940.7A Active CN115147838B (en) | 2022-06-30 | 2022-06-30 | Image processing method, device, vehicle, medium, and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115147838B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108152831A (en) * | 2017-12-06 | 2018-06-12 | 中国农业大学 | A kind of laser radar obstacle recognition method and system |
| CN109804223A (en) * | 2016-10-11 | 2019-05-24 | 御眼视觉技术有限公司 | Based on the barrier navigation vehicle detected |
| CN110929639A (en) * | 2019-11-20 | 2020-03-27 | 北京百度网讯科技有限公司 | Method, apparatus, device and medium for determining position of obstacle in image |
| WO2020206708A1 (en) * | 2019-04-09 | 2020-10-15 | 广州文远知行科技有限公司 | Obstacle recognition method and apparatus, computer device, and storage medium |
| WO2020259284A1 (en) * | 2019-06-27 | 2020-12-30 | 华为技术有限公司 | Obstacle detection method and device |
| CN113050063A (en) * | 2021-04-29 | 2021-06-29 | 深圳市利拓光电有限公司 | Obstacle detection method, device and equipment based on laser sensor and storage medium |
| CN113205088A (en) * | 2021-07-06 | 2021-08-03 | 禾多科技（北京）有限公司 | Obstacle image presentation method, electronic device, and computer-readable medium |
| CN113837305A (en) * | 2021-09-29 | 2021-12-24 | 北京百度网讯科技有限公司 | Target detection and model training method, device, equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10204279B2 (en) * | 2015-07-30 | 2019-02-12 | Magna Electronics Inc. | Vehicle vision system with object detection |
-
2022
- 2022-06-30 CN CN202210772940.7A patent/CN115147838B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109804223A (en) * | 2016-10-11 | 2019-05-24 | 御眼视觉技术有限公司 | Based on the barrier navigation vehicle detected |
| CN108152831A (en) * | 2017-12-06 | 2018-06-12 | 中国农业大学 | A kind of laser radar obstacle recognition method and system |
| WO2020206708A1 (en) * | 2019-04-09 | 2020-10-15 | 广州文远知行科技有限公司 | Obstacle recognition method and apparatus, computer device, and storage medium |
| WO2020259284A1 (en) * | 2019-06-27 | 2020-12-30 | 华为技术有限公司 | Obstacle detection method and device |
| CN110929639A (en) * | 2019-11-20 | 2020-03-27 | 北京百度网讯科技有限公司 | Method, apparatus, device and medium for determining position of obstacle in image |
| CN113050063A (en) * | 2021-04-29 | 2021-06-29 | 深圳市利拓光电有限公司 | Obstacle detection method, device and equipment based on laser sensor and storage medium |
| CN113205088A (en) * | 2021-07-06 | 2021-08-03 | 禾多科技（北京）有限公司 | Obstacle image presentation method, electronic device, and computer-readable medium |
| CN113837305A (en) * | 2021-09-29 | 2021-12-24 | 北京百度网讯科技有限公司 | Target detection and model training method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115147838A (en) | 2022-10-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109668742B (en) | Laser radar-based unmanned vehicle testing method and device | |
| US10917704B1 (en) | Automated video preview generation | |
| EP3176709A1 (en) | Video categorization method and apparatus, computer program and recording medium | |
| CN104918107A (en) | Video file identification processing method and device | |
| CN111651263A (en) | Resource processing method and device of mobile terminal, computer equipment and storage medium | |
| US20200372911A1 (en) | Speech recognition device and method | |
| CN104951335A (en) | Processing method and device for application program installation packages | |
| US11869537B1 (en) | Language agnostic automated voice activity detection | |
| CN106227419A (en) | Screenshotss method and device | |
| US20220124646A1 (en) | Method and apparatus for sending synchronization signal | |
| CN109858558A (en) | Training method, device, electronic equipment and the storage medium of disaggregated model | |
| EP3901827A1 (en) | Image processing method and apparatus based on super network, intelligent device and computer storage medium | |
| CN106228836A (en) | Indoor parking guide method and device | |
| CN104679386B (en) | The method and apparatus of recording processing | |
| CN105094830A (en) | Method and device for executing target functions | |
| CN107820684A (en) | Channel Detection, method for sending information, device and communication equipment | |
| CN113610811B (en) | Automobile instrument panel indicator lamp testing method and device and storage medium | |
| CN105513045A (en) | Image processing method, device and terminal | |
| CN109688533B (en) | Wearable device positioning method and device and storage medium | |
| CN105516595A (en) | Shooting method and device | |
| US20200007772A1 (en) | Imaging reproducing method and apparatus | |
| CN115147838B (en) | Image processing method, device, vehicle, medium, and program product | |
| CN116611482B (en) | Model training method, device, electronic equipment and medium | |
| CN104899151B (en) | Internal storage data reading/writing method and device | |
| CN105589575A (en) | Input method invoking method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |