CN112597960A - Image processing method, image processing device and computer readable storage medium - Google Patents
Image processing method, image processing device and computer readable storage medium Download PDFInfo
- Publication number
- CN112597960A CN112597960A CN202011617609.5A CN202011617609A CN112597960A CN 112597960 A CN112597960 A CN 112597960A CN 202011617609 A CN202011617609 A CN 202011617609A CN 112597960 A CN112597960 A CN 112597960A
- Authority
- CN
- China
- Prior art keywords
- image
- road traffic
- sub
- images
- traffic target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 51
- 238000003672 processing method Methods 0.000 title claims abstract description 22
- 238000003062 neural network model Methods 0.000 claims abstract description 27
- 238000000034 method Methods 0.000 claims description 38
- 238000001514 detection method Methods 0.000 description 16
- 238000013528 artificial neural network Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 10
- 230000005540 biological transmission Effects 0.000 description 9
- 238000004590 computer program Methods 0.000 description 8
- PCTMTFRHKVHKIS-BMFZQQSSSA-N (1s,3r,4e,6e,8e,10e,12e,14e,16e,18s,19r,20r,21s,25r,27r,30r,31r,33s,35r,37s,38r)-3-[(2r,3s,4s,5s,6r)-4-amino-3,5-dihydroxy-6-methyloxan-2-yl]oxy-19,25,27,30,31,33,35,37-octahydroxy-18,20,21-trimethyl-23-oxo-22,39-dioxabicyclo[33.3.1]nonatriaconta-4,6,8,10 Chemical group C1C=C2C[C@@H](OS(O)(=O)=O)CC[C@]2(C)[C@@H]2[C@@H]1[C@@H]1CC[C@H]([C@H](C)CCCC(C)C)[C@@]1(C)CC2.O[C@H]1[C@@H](N)[C@H](O)[C@@H](C)O[C@H]1O[C@H]1/C=C/C=C/C=C/C=C/C=C/C=C/C=C/[C@H](C)[C@@H](O)[C@@H](C)[C@H](C)OC(=O)C[C@H](O)C[C@H](O)CC[C@@H](O)[C@H](O)C[C@H](O)C[C@](O)(C[C@H](O)[C@H]2C(O)=O)O[C@H]2C1 PCTMTFRHKVHKIS-BMFZQQSSSA-N 0.000 description 7
- 230000008447 perception Effects 0.000 description 7
- 238000013135 deep learning Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 239000002699 waste material Substances 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000008094 contradictory effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
The application provides an image processing method, an image processing device and a computer readable storage medium. The image processing method comprises the following steps: acquiring an image to be processed, wherein the resolution of the image is a first resolution; dividing the image into a plurality of sub-images, wherein the resolution of each sub-image is a second resolution, and the second resolution is smaller than the first resolution; and inputting the plurality of sub-images into a neural network model for parallel operation, and identifying the road traffic target in the sub-images. According to the scheme, the image to be processed is firstly acquired, then the image to be processed is divided into a plurality of sub-images, the plurality of sub-images are simultaneously input into the neural network model for parallel operation, although the resolution of the sub-images is smaller than that of the image to be processed, all the sub-images comprise all the characteristics of the image to be processed, and the neural network model performs parallel operation on the plurality of sub-images, so that the resolution is guaranteed, and the real-time performance of image processing is guaranteed.
Description
Technical Field
The present application relates to the field of automatic driving image recognition, and in particular, to an image processing method, an image processing apparatus, a computer-readable storage medium, and a processor.
Background
In recent years, in the aspect of target identification and detection of camera images, the performance is greatly improved by means of a deep learning neural network technology. Therefore, the deep learning target detection technology is gradually applied to the target detection of the automatic driving camera perception system. In the application of the deep learning neural network in target detection, data training is very important work except for the design of a network model, and weight parameters of tens of millions or even hundreds of millions of neurons on the trained network model can be obtained through the training of mass data. The data training quality determines the accuracy of the neural network when used for vehicle-mounted real-time detection.
The applicant researches and discovers in the field of automatic driving camera image target perception through a target detection deep learning neural network model, and the contradiction and the disadvantages exist in the aspect of data processing as follows:
1. contradiction between resolution and real-time:
due to the complex traffic scene, the targets are always randomly distributed from near to far and from left to right, and the sizes and the types of the targets are also randomly distributed. Therefore, to achieve better detection accuracy in the aspect of automatic driving perception, it is necessary that the resolution of the image output by the camera is as high as possible, so that a target far away from the vehicle can be accurately detected. However, due to the limited bandwidth and computing power of the vehicle-mounted transmission and computing device, the resolution of the image output by the camera cannot be increased at once, which may result in occupying the transmission bandwidth of other sensors and increase the time consumed by the vehicle-mounted computing device to process one image by a square multiple, resulting in that the automatic driving perception processing cannot meet the required real-time requirement (the processing frequency of 30HZ is generally required for the camera).
2. Resource waste: in order to ensure a long field of view, the camera is generally mounted on the vehicle at an angle close to the horizontal shooting angle, and the upper half of the shot image usually contains a large amount of sky area. And the real targets to be detected are basically distributed in the lower half part of the image. Therefore, the image data is equivalent to the image data of the upper half part, and precious resources of vehicle-mounted transmission and computing equipment are wasted.
Disclosure of Invention
The present application mainly aims to provide an image processing method, an image processing apparatus, a computer-readable storage medium, and a processor, so as to solve the problem in the prior art that the resolution and the real-time performance of an image are contradictory in terms of automatic driving perception.
In order to achieve the above object, according to an aspect of the present application, there is provided an image processing method including: acquiring an image to be processed, wherein the resolution of the image is a first resolution; dividing the image into a plurality of sub-images, wherein the resolution of each sub-image is a second resolution, and the second resolution is smaller than the first resolution; and inputting the plurality of sub-images into a neural network model for parallel operation, and identifying the road traffic target in the sub-images.
Further, after the image is divided into a plurality of sub-images and before the plurality of sub-images are input into a neural network model for parallel operation and the road traffic target in the sub-images is identified, the method further comprises the following steps: and discarding part of the sub-images in the sub-images, wherein the discarded sub-images do not comprise the road traffic target.
Further, the method further includes overlapping portions of any two adjacent sub-images to form an overlapping region.
Further, the method further comprises: determining whether the road traffic target is within the overlap region; and if the road traffic target is in the overlapping area, performing overlapping processing on the road traffic target.
Further, the overlapping processing of the road traffic target includes: defining two adjacent sub-images as a left image and a right image respectively; acquiring a first boundary frame, wherein the first boundary frame is a boundary frame of a part of the road traffic target in the left image; acquiring a second boundary frame, wherein the second boundary frame is a boundary frame of a part of the road traffic target in the right image; determining whether the road traffic target in the left image and the road traffic target in the right image are the same road traffic target at least according to an area sum and an overlapping area, wherein the area sum is the sum of the area of the first boundary frame and the area of the second boundary frame, and the overlapping area is the area of the overlapping part of the first boundary frame and the second boundary frame.
Further, determining whether the road traffic target in the left image and the road traffic target in the right image are the same road traffic target at least according to the area and the overlapping area, includes: determining that the road traffic target in the left image and the road traffic target in the right image are the same road traffic target when the area ratio is larger than or equal to a predetermined value, wherein the area ratio is the ratio of the overlapping area to the area sum; determining that the road traffic target in the left image and the road traffic target in the right image are not the same road traffic target if the area ratio is less than the predetermined value.
Further, determining whether the road traffic target is within the overlap region comprises: obtaining coordinates of four vertexes of the first bounding box; acquiring coordinates of four vertexes of the second bounding box; and determining whether the road traffic target is in the overlapping area according to the coordinates of the four vertexes of the first boundary box, the coordinates of the four vertexes of the second boundary box, the size of the sub-image and the size of the overlapping area.
According to another aspect of the present application, there is provided an image processing apparatus including: the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring an image to be processed, and the resolution of the image is a first resolution; a dividing unit, configured to divide the image into a plurality of sub-images, where a resolution of each of the sub-images is a second resolution, and the second resolution is smaller than the first resolution; and the operation unit is used for inputting the sub-images into a neural network model for parallel operation and identifying the road traffic target in the sub-images.
According to still another aspect of the present application, there is provided a computer-readable storage medium including a stored program, wherein the program, when executed, controls an apparatus in which the computer-readable storage medium is located to perform any one of the image processing methods.
According to yet another aspect of the present application, there is provided a processor for executing a program, wherein the program executes any one of the image processing methods.
By applying the technical scheme of the application, the image to be processed is firstly acquired, then the image to be processed is divided into a plurality of sub-images, and then the plurality of sub-images are simultaneously input into the neural network model for parallel operation.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:
FIG. 1 shows a flow diagram of a method of processing an image according to an embodiment of the application;
FIG. 2 shows a schematic diagram of an overlap process for road traffic targets according to an embodiment of the application;
fig. 3 shows a schematic diagram of an image processing apparatus according to an embodiment of the application.
Detailed Description
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements may also be present. Also, in the specification and claims, when an element is described as being "connected" to another element, the element may be "directly connected" to the other element or "connected" to the other element through a third element.
As described in the background art, in order to solve the above problem that the resolution and the real-time performance of the image in the aspect of the automatic driving perception are contradictory in the prior art, embodiments of the present application provide a processing method, a processing apparatus, a computer-readable storage medium, and a processor for the image.
According to an embodiment of the present application, there is provided a method of processing an image.
Fig. 1 is a flowchart of a method of processing an image according to an embodiment of the present application. As shown in fig. 1, the method comprises the steps of:
step S101, acquiring an image to be processed, wherein the resolution of the image is a first resolution;
step S102, dividing the image into a plurality of sub-images, wherein the resolution of each sub-image is a second resolution, and the second resolution is smaller than the first resolution;
step S103, inputting a plurality of sub-images into a neural network model for parallel operation, and identifying the road traffic target in the sub-images.
Specifically, inputting a plurality of the sub-images into a neural network model for parallel operation includes: and (3) calling a parallel data processing mode (or a CPU multi-process mode) of the GPU by an algorithm system to input the data into the neural network for parallel processing.
Specifically, the image to be processed is obtained by a camera mounted on the autonomous vehicle, and in order to ensure higher detection accuracy, the resolution of the image captured by the camera is higher, but the image with higher resolution may occupy the transmission bandwidth of other sensors, and the time consumed by the onboard computing device to process one image is increased by a square multiple, so that the autonomous driving perception processing cannot meet the required real-time requirement.
Specifically, assume that the resolution of the raw output image of the camera mounted on the autonomous vehicle is: 2048 × 1536 (aspect ratio of 4:3), the resolution (i.e., size) requirements of the deep learning neural network model for the input image are: 640 × 640, the original 2048 × 1536 image is reduced to 640 × 640 and input to the neural network operation. According to the scheme, 5 blocks of 640 x 640 sub-images can be cut from an original output image, 5 blocks of 640 x 640 sub-images can be cut from the lower edge of the original output image, the images are divided into two layers, the first layer is 2 blocks, and the images are laid in the middle of the images; the second layer is 3 blocks and is laid out at the lower end of the image.
According to the scheme, the image to be processed is obtained firstly, then the image to be processed is divided into a plurality of sub-images, the plurality of sub-images are simultaneously input into the neural network model to be subjected to parallel operation, although the resolution of the sub-images is smaller than that of the image to be processed, all the sub-images comprise all the characteristics of the image to be processed, and the neural network model performs parallel operation on the plurality of sub-images, so that the resolution is guaranteed, and the real-time performance of image processing is guaranteed.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
In an embodiment of the present application, after the image is divided into a plurality of sub-images, and before the plurality of sub-images are input into a neural network model for parallel operation to identify the road traffic target in the sub-images, the method further includes: and discarding part of the sub-images in the sub-images, wherein the discarded sub-images do not comprise the road traffic target. In general, in order to secure a long field of view, a camera is generally mounted on a vehicle at an angle close to a horizontal photographing angle, and the upper half of a photographed image generally includes a large amount of sky area. The real road traffic targets to be detected are basically distributed on the lower half part of the image. Therefore, the image data equivalent to the upper half part wastes valuable resources of vehicle-mounted transmission and computing equipment, the sub-images which do not comprise the road traffic target are abandoned, the valuable resources of the vehicle-mounted transmission and computing equipment are saved, the image processing speed is increased due to the abandonment of a part of the sub-images, and the resolution of the image to be processed is indirectly reduced. That is to say, the method of the invention not only keeps the resolution of the image input to the neural network operation, but also abandons the resource waste caused by the operation of the invalid area, and the real-time performance is not affected by the parallel computing mode. Therefore, the accuracy of the target detection of the neural network is greatly improved.
In an embodiment of the present application, in order to avoid losing the small object located near the boundary of the sub-images, and enable the small object to have a more complete shape in at least one of the sub-images, the method further includes overlapping portions of any two adjacent sub-images to form an overlapping region. Specifically, the adjacent boundaries of any two adjacent sub-images may be made to have an overlap of 30 pixel width. By adopting the existing scheme, the width and the height of the image are reduced, and for some road traffic targets with smaller areas, the image is reduced, so that the characteristic loss is serious, and the correct detection of an image recognition algorithm is not facilitated. According to the scheme, the original image is only segmented, and the characteristics of the road traffic target with a small area are not lost.
In an embodiment of the present application, the method further includes: determining whether the road traffic target is within the overlap area; and when the road traffic target is in the overlapping area, performing overlapping processing on the road traffic target. That is, in the case that it is determined that the road traffic target is in the overlapping area, the road traffic target needs to be overlapped to realize accurate identification of the road traffic target.
In an embodiment of the present application, as shown in fig. 2, the performing an overlapping process on the road traffic target includes: defining two adjacent sub-images as a left image and a right image respectively; acquiring a first boundary frame, wherein the first boundary frame is a boundary frame of a part of the road traffic target in the left image; acquiring a second boundary frame, wherein the second boundary frame is a boundary frame of a part of the road traffic target in the right image; determining whether the road traffic target in the left image and the road traffic target in the right image are the same road traffic target at least based on an area sum of an area of the first bounding box and an area of the second bounding box and an overlapping area of an overlapping portion of the first bounding box and the second bounding box. Since there may be a plurality of road traffic targets in the overlapping area, in order to realize accurate identification of the road traffic target, it is necessary to determine whether the road traffic target located in the left image and the road traffic target located in the right image are the same road traffic target, so as to realize accurate identification of the road traffic target. The preferred way to determine whether the road traffic target in the left image and the road traffic target in the right image are the same road traffic target in the overlapping area of two adjacent sub-images is to pass the area and the overlapping area.
In an embodiment of the present application, determining whether the road traffic target in the left image and the road traffic target in the right image are the same road traffic target at least according to an area and an overlapping area includes: determining that the road traffic target in the left image and the road traffic target in the right image are the same road traffic target when the area ratio is greater than or equal to a predetermined value, the area ratio being a ratio of the overlap area to the area sum; and determining that the road traffic target in the left image and the road traffic target in the right image are not the same road traffic target when the area ratio is smaller than the predetermined value. Specifically, the predetermined value may be set to 0.5, and of course, a person skilled in the art may set an appropriate predetermined value according to actual circumstances.
In an embodiment of the present application, determining whether the road traffic target is within the overlap area includes: acquiring coordinates of four vertexes of the first bounding box; acquiring coordinates of four vertexes of the second bounding box; and determining whether the road traffic target is within the overlap area according to the coordinates of the four vertices of the first bounding box, the coordinates of the four vertices of the second bounding box, the size of the sub-image, and the size of the overlap area.
In a specific embodiment of the present application, the specific implementation manner of determining whether the road traffic target is in the overlap area is as follows: the boundary frame detected by the target car in the left image is ABCD (a is the upper left corner, B is the upper right corner, C is the lower right corner, and D is the lower left corner), and the boundary frame detected in the right image is EFGH (E is the upper left corner, F is the upper right corner, G is the lower right corner, and H is the lower left corner).
Firstly, judging whether the bounding box ABCD is wholly or partially in the overlapping area, wherein the judgment conditions are as follows:
b.x > (640-30) or C.x > (640-30)
Wherein the left image and the right image are both 640 x 640 in size, the width of the overlap area is 30, B.x represents the x-coordinate of point B, C.x represents the x-coordinate of point C, in the case of C.x > (640-30), the bounding box ABCD is described as being entirely within the overlap area, in the case of B.x > (640-30) and C.x < (640-30), the bounding box ABCD is described as being partially within the overlap area, and in the case of B.x < (640-30), the bounding box ABCD is described as not being within the overlap area;
similarly, it is determined whether the right image bounding box is in the same overlap region:
e.x <30 or H.x <30
Where E.x represents the x-coordinate of point E and H.x represents the x-coordinate of point H.
If the bounding box ABCD and the bounding box EFGH meet the conditions of the overlapping area, that is, the road traffic Object can be determined to be in the overlapping area, then whether the two bounding boxes represent the same Object is continuously judged, the invention designs the following calculation method to obtain the metric index Object _ one for judging whether the two bounding boxes represent the same Object:
Object_one＝Area_and/Area_all
where Area _ all is the Area of bounding box ABCD plus the Area of bounding box EFGH:
Area_all＝((B.x–A.x)×(C.y–B.y))+((F.x–E.x)×(G.y–E.y))
where Area _ and is the Area of the overlap of the two rectangles, A.x represents the x-coordinate of point A, B.x represents the x-coordinate of point B, B.y represents the y-coordinate of point C, B.y represents the y-coordinate of point C; f.x represents the x coordinate of point F, E.x represents the x coordinate of point E, G.y represents the y coordinate of point G, and E.y represents the y coordinate of point E;
the calculation method is as follows:
Area_and＝(min(B.x,F.x)–max(A.x,E.x))×(min(D.y,G.y)–max(A.y,E.y))
Area_and＝max(0,Area_and)
the final Object _ one result is:
Object_one＝Area_and/Area_all
the pointer Object _ one calculated by the above method will be a floating point number with a value ranging from 0 to 1, and we set the threshold T to 0.5:
if two bounding boxes in the overlapping area of two image blocks satisfy:
Object_one>T
then we consider this as a target and merge its bounding boxes to be output as the target detection result of the neural network.
According to the scheme, 5100 cameras are adopted to acquire road traffic images in real time, and the road traffic images comprise real street scenes, vehicles, pedestrians, bicycles, motorcycles, tricycles and the like. We adopt the yolo-v5 neural network model newly released in 2020 to perform image target recognition.
Target detection rate (in image set: total number of detected targets divided by total number of all labeled targets):
the original image is reduced to 640 multiplied by 640 to be input into a neural network, and the detection rate is 86.6 percent;
after the data are processed by the method and input into a neural network, the detection rate reaches 96.5 percent;
therefore, the data processing method greatly improves the detection rate of the automatic driving target from 86.6% to 96.5%. And some small targets which are far away are not detected by the conventional method, but can be detected by the method.
The embodiment of the present application further provides an image processing apparatus, and it should be noted that the image processing apparatus according to the embodiment of the present application may be used to execute the image processing method provided in the embodiment of the present application. The following describes an image processing apparatus according to an embodiment of the present application.
Fig. 3 is a schematic diagram of an image processing apparatus according to an embodiment of the present application. As shown in fig. 3, the apparatus includes:
an acquiring unit 10, configured to acquire an image to be processed, where a resolution of the image is a first resolution;
a dividing unit 20 configured to divide the image into a plurality of sub-images, wherein the resolution of each sub-image is a second resolution, and the second resolution is smaller than the first resolution;
and an operation unit 30, configured to input a plurality of sub-images into the neural network model to perform parallel operation, and identify a road traffic target in the sub-images.
In the scheme, the acquisition unit acquires an image to be processed, the segmentation unit segments the image to be processed into a plurality of sub-images, the operation unit simultaneously inputs the plurality of sub-images into the neural network model for parallel operation, although the resolution of the sub-images is smaller than that of the image to be processed, all the sub-images include all the characteristics of the image to be processed, and the neural network model performs parallel operation on the plurality of sub-images, so that the resolution is guaranteed, and the real-time performance of image processing is guaranteed.
In an embodiment of the application, the apparatus further includes a discarding unit, configured to discard a part of the sub-images in the discarded sub-images, after the image is divided into the plurality of sub-images, and before the plurality of sub-images are input into the neural network model to perform parallel operations to identify the road traffic target in the sub-images, where the discarded sub-images do not include the road traffic target. In general, in order to secure a long field of view, a camera is generally mounted on a vehicle at an angle close to a horizontal photographing angle, and the upper half of a photographed image generally includes a large amount of sky area. The real road traffic targets to be detected are basically distributed on the lower half part of the image. Therefore, the image data equivalent to the upper half part wastes valuable resources of vehicle-mounted transmission and computing equipment, the sub-images which do not comprise the road traffic target are abandoned, the valuable resources of the vehicle-mounted transmission and computing equipment are saved, the image processing speed is increased due to the abandonment of a part of the sub-images, and the resolution of the image to be processed is indirectly reduced. The scheme of the invention not only keeps the resolution of the image input to the neural network operation, but also abandons the resource waste caused by the operation of the invalid area, and the real-time performance is not influenced by the parallel computing mode. Therefore, the accuracy of the target detection of the neural network is greatly improved.
In an embodiment of the application, in order to avoid losing the small object located near the boundary of the sub-images, and enable the small object to have a more complete shape in at least one of the sub-images, the apparatus further includes an overlapping unit, and the overlapping unit is configured to overlap parts of any two adjacent sub-images to form an overlapping region. Specifically, the adjacent boundaries of any two adjacent sub-images may be made to have an overlap of 30 pixel width. By adopting the existing scheme, the width and the height of the image are reduced, and for some road traffic targets with smaller areas, the image is reduced, so that the characteristic loss is serious, and the correct detection of an image recognition algorithm is not facilitated. According to the scheme, the original image is only segmented, and the characteristics of the road traffic target with a small area are not lost.
In an embodiment of the present application, the apparatus further includes a determining unit and a processing unit, the determining unit is configured to determine whether the road traffic target is within the overlapping area; the processing unit is used for performing overlapping processing on the road traffic target when the road traffic target is in the overlapping area. That is, in the case that it is determined that the road traffic target is in the overlapping area, the road traffic target needs to be overlapped to realize accurate identification of the road traffic target.
In an embodiment of the present application, as shown in fig. 2, the processing unit includes a defining module, a first obtaining module, a second obtaining module, and a determining module, where the defining module is configured to define two adjacent sub-images as a left image and a right image, respectively; the first acquisition module is used for acquiring a first boundary frame, wherein the first boundary frame is a partial boundary frame of the road traffic target in the left image; the second acquisition module is used for acquiring a second boundary frame, wherein the second boundary frame is a boundary frame of a part of the road traffic target in the right image; the determining module is configured to determine whether the road traffic target in the left image and the road traffic target in the right image are the same road traffic target at least according to an area sum and an overlapping area, where the area sum is a sum of an area of the first bounding box and an area of the second bounding box, and the overlapping area is an area of an overlapping portion of the first bounding box and the second bounding box. Since there may be a plurality of road traffic targets in the overlapping area, in order to realize accurate identification of the road traffic target, it is necessary to determine whether the road traffic target located in the left image and the road traffic target located in the right image are the same road traffic target, so as to realize accurate identification of the road traffic target. The preferred way to determine whether the road traffic target in the left image and the road traffic target in the right image are the same road traffic target in the overlapping area of two adjacent sub-images is to pass the area and the overlapping area.
In an embodiment of the application, the determining module is further configured to determine that the road traffic target in the left image and the road traffic target in the right image are the same road traffic target when an area ratio is greater than or equal to a predetermined value, where the area ratio is a ratio of the overlapping area to the area sum; and determining that the road traffic target in the left image and the road traffic target in the right image are not the same road traffic target when the area ratio is smaller than the predetermined value. Specifically, the predetermined value may be set to 0.5, and of course, a person skilled in the art may set an appropriate predetermined value according to actual circumstances.
In an embodiment of the application, the determining unit is further configured to obtain coordinates of four vertices of the first bounding box; acquiring coordinates of four vertexes of the second bounding box; and determining whether the road traffic target is within the overlap area according to the coordinates of the four vertices of the first bounding box, the coordinates of the four vertices of the second bounding box, the size of the sub-image, and the size of the overlap area.
The image processing device comprises a processor and a memory, wherein the acquisition unit, the segmentation unit, the operation unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. One or more than one kernel can be set, and the real-time performance of image processing is ensured while the resolution is ensured by adjusting kernel parameters.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
The embodiment of the invention provides a computer-readable storage medium, which comprises a stored program, wherein when the program runs, a device where the computer-readable storage medium is located is controlled to execute the image processing method.
The embodiment of the invention provides a processor, which is used for running a program, wherein the processing method of the image is executed when the program runs.
The embodiment of the invention provides equipment, which comprises a processor, a memory and a program which is stored on the memory and can run on the processor, wherein when the processor executes the program, at least the following steps are realized:
step S101, acquiring an image to be processed, wherein the resolution of the image is a first resolution;
step S102, dividing the image into a plurality of sub-images, wherein the resolution of each sub-image is a second resolution, and the second resolution is smaller than the first resolution;
step S103, inputting a plurality of sub-images into a neural network model for parallel operation, and identifying the road traffic target in the sub-images.
The device herein may be a server, a PC, a PAD, a mobile phone, etc.
The present application further provides a computer program product adapted to perform a program of initializing at least the following method steps when executed on a data processing device:
step S101, acquiring an image to be processed, wherein the resolution of the image is a first resolution;
step S102, dividing the image into a plurality of sub-images, wherein the resolution of each sub-image is a second resolution, and the second resolution is smaller than the first resolution;
step S103, inputting a plurality of sub-images into a neural network model for parallel operation, and identifying the road traffic target in the sub-images.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
From the above description, it can be seen that the above-described embodiments of the present application achieve the following technical effects:
1) the image processing method comprises the steps of firstly obtaining an image to be processed, then dividing the image to be processed into a plurality of sub-images, and simultaneously inputting the plurality of sub-images into the neural network model for parallel operation.
2) According to the image processing device, the acquisition unit acquires an image to be processed, the segmentation unit segments the image to be processed into a plurality of sub-images, the operation unit simultaneously inputs the plurality of sub-images into the neural network model for parallel operation, although the resolution of the sub-images is smaller than that of the image to be processed, all the sub-images include all features of the image to be processed, and the neural network model performs parallel operation on the plurality of sub-images, so that the resolution is guaranteed, and the real-time performance of image processing is guaranteed.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A method of processing an image, comprising:
acquiring an image to be processed, wherein the resolution of the image is a first resolution;
dividing the image into a plurality of sub-images, wherein the resolution of each sub-image is a second resolution, and the second resolution is smaller than the first resolution;
and inputting the plurality of sub-images into a neural network model for parallel operation, and identifying the road traffic target in the sub-images.
2. The processing method according to claim 1, wherein after the image is divided into a plurality of sub-images and before the plurality of sub-images are input into a neural network model for parallel operation to identify the road traffic target in the sub-images, the method further comprises:
and discarding part of the sub-images in the sub-images, wherein the discarded sub-images do not comprise the road traffic target.
3. The processing method according to claim 1 or 2, further comprising overlapping portions of any two adjacent sub-images to form an overlapping region.
4. The processing method according to claim 3, characterized in that the method further comprises:
determining whether the road traffic target is within the overlap region;
and if the road traffic target is in the overlapping area, performing overlapping processing on the road traffic target.
5. The processing method according to claim 4, wherein the performing of the overlapping processing on the road traffic target comprises:
defining two adjacent sub-images as a left image and a right image respectively;
acquiring a first boundary frame, wherein the first boundary frame is a boundary frame of a part of the road traffic target in the left image;
acquiring a second boundary frame, wherein the second boundary frame is a boundary frame of a part of the road traffic target in the right image;
determining whether the road traffic target in the left image and the road traffic target in the right image are the same road traffic target at least according to an area sum and an overlapping area, wherein the area sum is the sum of the area of the first boundary frame and the area of the second boundary frame, and the overlapping area is the area of the overlapping part of the first boundary frame and the second boundary frame.
6. The processing method of claim 5, wherein determining whether the road traffic target in the left image and the road traffic target in the right image are the same road traffic target based on at least an area and an overlap area comprises:
determining that the road traffic target in the left image and the road traffic target in the right image are the same road traffic target when the area ratio is larger than or equal to a predetermined value, wherein the area ratio is the ratio of the overlapping area to the area sum;
determining that the road traffic target in the left image and the road traffic target in the right image are not the same road traffic target if the area ratio is less than the predetermined value.
7. The processing method of claim 5, wherein determining whether the road traffic target is within the overlap region comprises:
obtaining coordinates of four vertexes of the first bounding box;
acquiring coordinates of four vertexes of the second bounding box;
and determining whether the road traffic target is in the overlapping area according to the coordinates of the four vertexes of the first boundary box, the coordinates of the four vertexes of the second boundary box, the size of the sub-image and the size of the overlapping area.
8. An apparatus for processing an image, comprising:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring an image to be processed, and the resolution of the image is a first resolution;
a dividing unit, configured to divide the image into a plurality of sub-images, where a resolution of each of the sub-images is a second resolution, and the second resolution is smaller than the first resolution;
and the operation unit is used for inputting the sub-images into a neural network model for parallel operation and identifying the road traffic target in the sub-images.
9. A computer-readable storage medium, comprising a stored program, wherein when the program runs, the computer-readable storage medium controls an apparatus to execute the image processing method according to any one of claims 1 to 7.
10. A processor, characterized in that the processor is configured to run a program, wherein the program is configured to execute the method of processing the image according to any one of claims 1 to 7 when running.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011617609.5A CN112597960A (en) | 2020-12-30 | 2020-12-30 | Image processing method, image processing device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011617609.5A CN112597960A (en) | 2020-12-30 | 2020-12-30 | Image processing method, image processing device and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112597960A true CN112597960A (en) | 2021-04-02 |
Family
ID=75206324
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011617609.5A Pending CN112597960A (en) | 2020-12-30 | 2020-12-30 | Image processing method, image processing device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112597960A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115019553A (en) * | 2021-07-22 | 2022-09-06 | 苏州旭安交通科技有限公司 | Pedestrian zebra crossing early warning device based on region detection |
-
2020
- 2020-12-30 CN CN202011617609.5A patent/CN112597960A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115019553A (en) * | 2021-07-22 | 2022-09-06 | 苏州旭安交通科技有限公司 | Pedestrian zebra crossing early warning device based on region detection |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113468967B (en) | Attention mechanism-based lane line detection method, attention mechanism-based lane line detection device, attention mechanism-based lane line detection equipment and attention mechanism-based lane line detection medium | |
| CN111311709B (en) | Method and device for generating high-precision map | |
| JP6313646B2 (en) | External recognition device | |
| KR101848019B1 (en) | Method and Apparatus for Detecting Vehicle License Plate by Detecting Vehicle Area | |
| CN109583393B (en) | Lane line end point identification method and device, equipment and medium | |
| WO2013186662A1 (en) | Multi-cue object detection and analysis | |
| CN111027535A (en) | License plate recognition method and related equipment | |
| CN111091023A (en) | Vehicle detection method and device and electronic equipment | |
| JP7119197B2 (en) | Lane attribute detection | |
| CN112991349B (en) | Image processing method, device, equipment and storage medium | |
| CN109102026A (en) | A kind of vehicle image detection method, apparatus and system | |
| CN112597960A (en) | Image processing method, image processing device and computer readable storage medium | |
| CN111598088A (en) | Target detection method and device, computer equipment and readable storage medium | |
| CN109800684A (en) | The determination method and device of object in a kind of video | |
| CN112784675B (en) | Target detection method and device, storage medium and terminal | |
| CN113763412A (en) | Image processing method and device, electronic equipment and computer readable storage medium | |
| CN111292331B (en) | Image processing method and device | |
| CN112818968A (en) | Target object classification method and device | |
| CN112785595A (en) | Target attribute detection, neural network training and intelligent driving method and device | |
| CN114494398B (en) | Processing method and device of inclined target, storage medium and processor | |
| CN113031010B (en) | Method, apparatus, computer readable storage medium and processor for detecting weather | |
| CN114066958A (en) | Method and device for predicting depth information of target, electronic device and storage medium | |
| CN112529943B (en) | Object detection method, object detection device and intelligent equipment | |
| CN114037976A (en) | Road traffic sign identification method and device | |
| CN113158864A (en) | Method and device for determining included angle between truck head and trailer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |