CN117237393A - Image processing method and device based on streaming media rearview mirror and computer equipment - Google Patents
Image processing method and device based on streaming media rearview mirror and computer equipment Download PDFInfo
- Publication number
- CN117237393A CN117237393A CN202311462899.4A CN202311462899A CN117237393A CN 117237393 A CN117237393 A CN 117237393A CN 202311462899 A CN202311462899 A CN 202311462899A CN 117237393 A CN117237393 A CN 117237393A
- Authority
- CN
- China
- Prior art keywords
- rearview
- space
- image
- dimensional
- rearview mirror
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 16
- 230000002159 abnormal effect Effects 0.000 claims abstract description 85
- 238000012545 processing Methods 0.000 claims abstract description 66
- 238000010586 diagram Methods 0.000 claims abstract description 56
- 238000000034 method Methods 0.000 claims abstract description 52
- 230000005540 biological transmission Effects 0.000 claims abstract description 38
- 238000003062 neural network model Methods 0.000 claims abstract description 17
- 230000000875 corresponding effect Effects 0.000 claims description 18
- 230000000295 complement effect Effects 0.000 claims description 14
- 238000000926 separation method Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 12
- 230000005856 abnormality Effects 0.000 claims description 8
- 238000011176 pooling Methods 0.000 claims description 8
- 239000011159 matrix material Substances 0.000 claims description 7
- 230000000007 visual effect Effects 0.000 claims description 7
- 239000000428 dust Substances 0.000 claims description 6
- 230000011218 segmentation Effects 0.000 claims description 6
- 230000002596 correlated effect Effects 0.000 claims description 4
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 6
- 238000013527 convolutional neural network Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000003708 edge detection Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Image Processing (AREA)
Abstract
The invention relates to an image processing method, a device and computer equipment based on a streaming media rearview mirror, wherein the method comprises the following steps: collecting a plurality of rearview mirror images through a streaming media rearview mirror, wherein the plurality of rearview mirror images at least comprise a left rearview mirror image and a right rearview mirror image; inputting a plurality of rearview mirror images into a four-dimensional streaming media modeling framework through corresponding data transmission channels respectively so as to generate a rearview space four-dimensional diagram through the four-dimensional streaming media modeling framework; inputting the four-dimensional image of the rearview space into an image convolution neural network model, and calibrating abnormal factors in the four-dimensional image of the rearview space through the image convolution neural network model; performing definition processing on the rear-view space four-dimensional map calibrated with the abnormal factors through an image processing algorithm to generate a display image associated with the rear-view space four-dimensional map, and displaying the display image; the images are identified, marked, filled and repaired through the pre-trained image convolution model, so that the effect of recovering the images to the optimal state is achieved.
Description
Technical Field
The present invention relates to the field of vehicle-mounted image processing technologies, and in particular, to an image processing method and apparatus based on a streaming media rearview mirror, and a computer device.
Background
Conventional automotive rearview mirrors typically employ a mirror or camera as an image capturing device. However, these devices suffer from a number of disadvantages. For example, the mirror is easily disturbed by the night lights and strong sunlight, thereby affecting the driver's vision; cameras require complex wiring and control systems to be installed and often fail to function properly in severe weather conditions.
In order to solve these problems, some new rearview mirror designs have emerged in recent years. One of the more advanced designs is the rearview mirror based on streaming technology. The rear view mirror converts the real-time video signal at the rear of the vehicle into a digital signal and transmits the digital signal to a display in the cockpit through a wireless network, so that a driver can more clearly observe the condition at the rear of the vehicle. In addition, the design has the advantages of simplicity, easiness in use, space saving and the like.
However, current streaming-based rearview mirrors have some problems. For example, since a certain time is required for transmission of a video signal, there is a certain delay in operations such as fast lane changing and wire combining, and thus judgment and reaction of a driver are affected. In addition, because the images behind the vehicle are often complex, shielding objects such as raindrops and dust are easy to appear, thereby influencing the definition and the visibility of the images.
Disclosure of Invention
The invention mainly aims to provide an image processing method, an image processing device and computer equipment based on a streaming media rearview mirror, which are used for identifying shielding objects such as raindrops and dust in an acquired image through a pre-trained image convolution model, marking the positions of the shielding objects, filling and repairing the marked shielding objects through an image processing algorithm of a layered channel, and achieving the effect of recovering the image to an optimal state.
In order to achieve the above object, the present invention provides an image processing method based on a streaming media rearview mirror, comprising the following steps:
collecting a plurality of rearview mirror images through a streaming media rearview mirror, wherein the plurality of rearview mirror images at least comprise a left rearview mirror image and a right rearview mirror image;
inputting a plurality of rearview mirror images into a four-dimensional streaming media modeling framework through corresponding data transmission channels respectively so as to generate a rearview space four-dimensional diagram through the four-dimensional streaming media modeling framework;
inputting the rearview space four-dimensional graph into an image convolution neural network model, and calibrating abnormal factors in the rearview space four-dimensional graph through the image convolution neural network model;
and performing sharpening processing on the rearview space four-dimensional map calibrated with the abnormal factors through an image processing algorithm to generate a display image associated with the rearview space four-dimensional map, and displaying the display image.
Further, the step of inputting the plurality of rearview mirror images into a four-dimensional streaming media modeling framework through corresponding data transmission channels to generate a rearview space four-dimensional map through the four-dimensional streaming media modeling framework comprises the following steps:
setting a corresponding number of data transmission channels according to the specific number of the rearview mirror images;
identifying and marking the shooting directions of a plurality of rearview mirror images;
the method comprises inputting several rearview mirror images marked with camera orientation into a four-dimensional stream media modeling framework to generate a rearview space four-dimensional map by the four-dimensional stream media modeling framework,
image stitching is carried out on a plurality of rearview mirror images according to the shooting direction marks;
identifying image edges of a plurality of rearview mirror images, and judging whether edge superposition occurs in image stitching of the plurality of rearview mirror images according to the image edges;
if the edges are overlapped, performing de-overlapping treatment after image splicing to generate a rearview space four-dimensional image;
and if the edges are not overlapped, carrying out broken line fitting processing after image splicing to generate a rearview space four-dimensional graph.
Further, the method for creating the four-dimensional streaming media modeling framework comprises the following steps:
creating a vehicle-mounted streaming media coordinate system xyr, wherein the vehicle-mounted streaming media coordinate system xyr comprises a driving direction x, a rearview viewing angle y and a steering inclination angle r;
and establishing a data transmission protocol between the vehicle-mounted streaming media coordinate system xyr and a data transmission channel so as to correlate the marking information t pre-correlated with the data transmission channel with the rearview viewing angle y to create the four-dimensional streaming media modeling framework xyrt.
Further, the step of inputting the four-dimensional view of the rearview space into an image convolutional neural network model to calibrate the abnormal factors in the four-dimensional view of the rearview space through the image convolutional neural network model comprises the following steps:
the method for calibrating the abnormal factors of the four-dimensional image of the rear-view space by the image convolutional neural network model comprises the following steps of,
continuously acquiring a rearview space four-dimensional map through an input layer, and importing the rearview space four-dimensional map into a convolution layer;
performing distance matrix segmentation processing on the rear-view space four-dimensional map through a convolution layer to perform matrix segmentation on front, middle and rear sections of the rear-view space four-dimensional map to generate a front section rear-view space map, a middle section rear-view space map and a rear section rear-view space map, and inputting the front section rear-view space map, the middle section rear-view space map and the rear section rear-view space map to a pooling layer;
the front section rearview space diagram, the middle section rearview space diagram and the rear section rearview space diagram are subjected to visual sharpening enhancement treatment through the pooling layer and output to the full-connection layer;
respectively matching abnormal factors in a front section rearview space diagram, a middle section rearview space diagram and a rear section rearview space diagram through a pre-trained factor template in the full-connection layer, and inputting the abnormal factors into the softmax layer, wherein the abnormal factors comprise, but are not limited to, water drops and dust;
the similarity judging process of the abnormal factors and the factor templates is carried out through the softmax layer, and when the similarity of the abnormal factors and the factor templates is judged to be larger than a set value, the abnormal factors are calibrated at the positions of the front section rearview space diagram, the middle section rearview space diagram and the rear section rearview space diagram;
and outputting the front section rearview space diagram, the middle section rearview space diagram and the rear section rearview space diagram calibrated with the abnormal factors by an output layer.
Further, the step of performing a sharpening process on the rearview space four-dimensional map calibrated with the anomaly factors by an image processing algorithm to generate a display image associated with the rearview space four-dimensional map and displaying the display image includes:
identifying an abnormal part which is shielded by an abnormal factor and a normal part which is not shielded in the rearview space four-dimensional diagram;
sharpening the fuzzy line between the abnormal part and the normal part to generate a separation line;
and carrying out the sharpening process on the abnormal part determined by the separation line according to the type of the abnormal factor.
Further, the step of performing a sharpening process on the abnormal portion determined by the separation line according to the type of the abnormality factor includes:
identifying perspective clue data of the rearview space four-dimensional graph which is shielded by the anomaly factors;
and importing the perspective clue data into an image processing algorithm for element complementation, generating first complement data and loading the first complement data into an abnormal part of the four-dimensional graph of the rearview space.
Further, the step of performing a sharpening process on the abnormal portion determined by the separation line according to the type of the abnormality factor includes:
the deduction data of the rearview space four-dimensional graph which is completely blocked by the abnormal factors are identified;
and importing the deduction data into an image processing algorithm for element completion, generating second completion data and loading the second completion data into an abnormal part of the four-dimensional map of the rearview space.
The invention relates to an image processing device based on a streaming media rearview mirror, which comprises:
the system comprises an acquisition unit, a control unit and a control unit, wherein the acquisition unit is used for acquiring a plurality of rearview mirror images through a streaming media rearview mirror, and the plurality of rearview mirror images at least comprise a left rearview mirror image and a right rearview mirror image;
the composition unit is used for inputting a plurality of rearview mirror images into a four-dimensional streaming media modeling framework through corresponding data transmission channels respectively so as to generate a rearview space four-dimensional diagram through the four-dimensional streaming media modeling framework;
the calibration unit is used for inputting the rearview space four-dimensional graph into an image convolution neural network model so as to calibrate abnormal factors in the rearview space four-dimensional graph through the image convolution neural network model;
and the processing unit is used for carrying out the sharpening processing on the rearview space four-dimensional graph calibrated with the abnormal factors through an image processing algorithm so as to generate a display image associated with the rearview space four-dimensional graph and display the display image.
The invention also provides a computer device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the steps of the image processing method based on the streaming media rearview mirror when executing the computer program.
The present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the streaming rearview mirror-based image processing method as described in any one of the above.
The image processing method and device based on the streaming media rearview mirror and the computer equipment provided by the invention have the following beneficial effects:
(1) The quality and effect of rearview mirror image processing are improved: by introducing a four-dimensional streaming media modeling frame and a convolutional neural network model and a method for performing the sharpening process on the four-dimensional map of the rearview space calibrated with the anomaly factors, the definition and the accuracy of the rearview mirror image can be effectively improved.
(2) Reducing image transmission bandwidth and delay: by utilizing a data compression algorithm and a quick data transmission channel based on UDP protocol, the data transmission bandwidth and delay can be reduced while the image quality is ensured, and the instantaneity and stability of the system are improved.
(3) The degree of automation of image processing is improved: by utilizing the deep learning algorithm to automatically identify and mark the shooting direction of the rearview mirror image, the manual intervention can be reduced, the degree of automation of image processing is improved, and the system operation cost is reduced.
Drawings
FIG. 1 is a schematic diagram illustrating steps of an image processing method based on a streaming rearview mirror according to an embodiment of the invention;
FIG. 2 is a block diagram of an image processing apparatus based on a streaming rearview mirror according to an embodiment of the invention;
fig. 3 is a block diagram schematically illustrating a structure of a computer device according to an embodiment of the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1, a flow chart of an image processing method based on a streaming media rearview mirror according to the present invention includes the following steps:
s1, collecting a plurality of rearview mirror images through a streaming media rearview mirror, wherein the plurality of rearview mirror images at least comprise a left rearview mirror image and a right rearview mirror image;
in the specific implementation process, with the development of the prior art, the number of camera units configured on a vehicle is increased gradually to ensure driving safety, and a plurality of rearview mirror images are acquired through the camera units.
S2, inputting a plurality of rearview mirror images into a four-dimensional streaming media modeling framework through corresponding data transmission channels respectively so as to generate a rearview space four-dimensional diagram through the four-dimensional streaming media modeling framework;
the specific implementation process of S2 is as follows:
setting a corresponding number of data transmission channels according to the specific number of the rearview mirror images; in streaming rearview mirror based image processing systems, the data transmission channel is an important channel for transmitting the rearview mirror image from the vehicle to a remote server or other terminal device. In order to ensure the real-time performance and stability of image transmission, a corresponding number of data transmission channels need to be provided. In particular, the specific number of said mirror images is determined according to the number of mirrors used by the vehicle, each mirror requiring a corresponding data transmission channel. For example, if there are two rearview mirrors on one vehicle, two data transmission channels need to be provided to transmit two rearview mirror images, respectively.
Identifying and marking the shooting directions of a plurality of rearview mirror images; in streaming rearview mirror image processing systems, several rearview mirror images may have different camera orientations, which can have an impact on subsequent image processing and analysis. Therefore, it is necessary to identify and mark the imaging orientation of each mirror image so that the system can perform corresponding processing according to the orientation thereof. For example, if the imaging direction of one rearview mirror image is left, then analysis and processing for the left lane are required at the time of lane shift detection; by identifying and marking the shooting directions of a plurality of rearview mirror images, the automation degree of image processing can be improved, manual intervention is reduced, the system operation cost is reduced, and more accurate basic data can be provided for subsequent image processing and analysis.
The method comprises inputting several rearview mirror images marked with camera orientation into a four-dimensional stream media modeling framework to generate a rearview space four-dimensional map by the four-dimensional stream media modeling framework,
image stitching is carried out on a plurality of rearview mirror images according to the shooting direction marks; image stitching of a plurality of rearview mirror images according to shooting direction marks means that images captured by a plurality of rearview mirror cameras are processed, firstly shooting directions (such as upper left and lower right) of each image are automatically recognized through a deep learning algorithm, and then images with the same direction are stitched according to time sequence to form a continuous video stream. By the aid of the method, the visual field range and coverage area of the rearview mirror image can be improved, a driver is helped to know the situation around the vehicle more comprehensively, and accordingly driving safety is improved.
Identifying image edges of a plurality of rearview mirror images, and judging whether edge superposition occurs in image stitching of the plurality of rearview mirror images according to the image edges; when image stitching is performed on a plurality of rearview mirror images, in order to ensure the accuracy and stability of stitching effect, the edge of each image needs to be identified, and whether the image stitching of the plurality of rearview mirror images is overlapped or not is judged according to the image edges. Specifically, edge detection is performed on each rearview mirror image through an edge detection algorithm (such as a Sobel algorithm, a Canny algorithm and the like) to obtain edge information in the image. Then, images of the same orientation are spliced in time sequence for images captured by the plurality of rearview mirror cameras. When the splicing operation is carried out, factors such as the overlapping degree of the splicing area, the brightness and the color consistency of the spliced image and the like are required to be considered, so that obvious edge overlapping phenomenon is avoided. In order to judge whether the image stitching of a plurality of rearview mirror images is overlapped or not, whether the edge stitching phenomenon exists or not can be determined by comparing the edge information of two adjacent stitched images. If the overlapping part of the edge information is found to exceed a certain threshold value, the image of the part is considered to have the edge overlapping phenomenon. At this time, corresponding processing measures such as readjusting the overlapping degree of the splicing region, adjusting the brightness and color consistency of the image, etc. are required to ensure that the splicing effect reaches the optimal state.
If the edges are overlapped, performing de-overlapping treatment after image splicing to generate a rearview space four-dimensional image; the de-registration process may be implemented by image fusion techniques. Firstly, aiming at the edge information superposition part of two adjacent spliced images, an image segmentation algorithm based on region growth can be adopted to separate the two adjacent spliced images, so that two independent image regions are formed. Then, the two image areas are subjected to image fusion to generate a new spliced image, and the image is free from edge overlapping.
And if the edges are not overlapped, carrying out broken line fitting processing after image splicing to generate a rearview space four-dimensional graph. When the edges of the rearview mirror images are not overlapped, image stitching can be performed, broken line fitting processing is performed, the part losing the visual field is deduced through clues, the deduced result is displayed in the rearview mirror images in a broken line mode, and finally a rearview space four-dimensional image is generated.
In another embodiment, the method for creating the four-dimensional streaming media modeling framework includes:
creating a vehicle-mounted streaming media coordinate system xyr, wherein the vehicle-mounted streaming media coordinate system xyr comprises a driving direction x, a rearview viewing angle y and a steering inclination angle r;
and establishing a data transmission protocol between the vehicle-mounted streaming media coordinate system xyr and a data transmission channel so as to correlate the marking information t pre-correlated with the data transmission channel with the rearview viewing angle y to create the four-dimensional streaming media modeling framework xyrt.
In the present embodiment, an in-vehicle streaming media coordinate system xyr is created: a three-dimensional coordinate system is established in which the x-axis represents the direction of travel, the y-axis represents the rear view angle, and r represents the steering angle of inclination. This coordinate system is used to describe the viewing direction and angle of view of the vehicle.
Establishing a data transmission channel and a protocol: the vehicle-mounted streaming media coordinate system xyr is associated with the data transmission channel. By establishing a data transfer protocol, the rearview mirror image and the marking information can be transferred from the streaming rearview mirror to a subsequent processing system.
Associating the marking information and the rearview angle: during data transmission, the data transmission channel is pre-associated with the tag information t and the rearview angle y. The tag information t generally indicates the occurrence of a particular event, such as a vehicle steering, braking, etc. The rearview viewing angle y represents viewing angle information in the vehicle rearview mirror collection image. By associating the marking information t with the rearview viewing angle y, the moment at which the marking information occurs and the corresponding viewing angle can be determined.
Creating a four-dimensional streaming media modeling framework xyrt: and combining the vehicle-mounted streaming media coordinate system xyr and the associated marking information t to form a four-dimensional streaming media modeling frame xyrt. This framework can be used for subsequent image processing and anomaly scaling.
S3, inputting the rearview space four-dimensional graph into an image convolution neural network model, and calibrating abnormal factors in the rearview space four-dimensional graph through the image convolution neural network model;
when S3 is specifically executed:
the method for calibrating the abnormal factors of the four-dimensional image of the rear-view space by the image convolutional neural network model comprises the following steps of,
continuously acquiring a rearview space four-dimensional map through an input layer, and importing the rearview space four-dimensional map into a convolution layer; in the running process of the vehicle, the image shot by the rearview mirror is changed continuously along with time, and in order to acquire and process the rearview space four-dimensional image in real time, a continuous mechanism needs to be arranged on an input layer, so that the system can continuously receive the latest rearview mirror image data and integrate the latest rearview mirror image data into a rearview space four-dimensional image.
Performing distance matrix segmentation processing on the rear-view space four-dimensional map through a convolution layer to perform matrix segmentation on front, middle and rear sections of the rear-view space four-dimensional map to generate a front section rear-view space map, a middle section rear-view space map and a rear section rear-view space map, and inputting the front section rear-view space map, the middle section rear-view space map and the rear section rear-view space map to a pooling layer; the range matrix refers to scene images of different ranges behind the vehicle, which are shot by the rearview mirror, and the rearview space four-dimensional image is divided into front, middle and rear three sections according to the range degree of the rear, so as to judge the importance degree (weight value) of the anomaly factors of the rear view space four-dimensional image when the softmax layer is formed.
The front section rearview space diagram, the middle section rearview space diagram and the rear section rearview space diagram are subjected to visual sharpening enhancement treatment through the pooling layer and output to the full-connection layer; and respectively carrying out pooling treatment on the front section, the middle section and the rear section rearview space map to obtain three feature maps, carrying out visual sharpening enhancement treatment on each feature map to enable the outline of an image to be clearer so as to improve the identification accuracy, respectively sending the three treated feature maps into a full-connection layer to carry out final image identification and treatment, carrying out visual sharpening enhancement treatment on the front section rearview space map, the middle section rearview space map and the rear section rearview space map through the pooling layer, and outputting the three treated feature maps to the full-connection layer, thereby effectively improving the image identification and the safety guarantee level in the vehicle driving process.
Respectively matching abnormal factors in a front section rearview space diagram, a middle section rearview space diagram and a rear section rearview space diagram through a pre-trained factor template in the full-connection layer, and inputting the abnormal factors into the softmax layer, wherein the abnormal factors comprise, but are not limited to, water drops and dust;
the similarity judging process of the abnormal factors and the factor templates is carried out through the softmax layer, and when the similarity of the abnormal factors and the factor templates is judged to be larger than a set value, the abnormal factors are calibrated at the positions of the front section rearview space diagram, the middle section rearview space diagram and the rear section rearview space diagram; the probability distribution obtained by the softmax layer is compared with a factor template, if the similarity is larger than a set value, the abnormal factor is considered to be matched with the factor template, and the abnormal factor is calibrated at the positions of the front section rearview space diagram, the middle section rearview space diagram and the rear section rearview space diagram,
And outputting the front section rearview space diagram, the middle section rearview space diagram and the rear section rearview space diagram calibrated with the abnormal factors by an output layer.
And S4, performing sharpening processing on the rearview space four-dimensional map calibrated with the abnormal factors through an image processing algorithm to generate a display image associated with the rearview space four-dimensional map, and displaying the display image.
Specifically executing S4:
identifying an abnormal part which is shielded by an abnormal factor and a normal part which is not shielded in the rearview space four-dimensional diagram;
sharpening the fuzzy line between the abnormal part and the normal part to generate a separation line;
and carrying out the sharpening process on the abnormal part determined by the separation line according to the type of the abnormal factor.
The first step is to identify the abnormal portion that is occluded by the abnormality factor and the normal portion that is not occluded. Then, after the blurred line between the abnormal portion and the normal portion is identified, sharpening processing is required to generate a separation line. Thus, the abnormal part and the normal part can be more clearly distinguished, and the subsequent processing is facilitated. And finally, carrying out the definition processing on the abnormal part determined by the separation line according to the type of the abnormal factor. Specifically, the abnormal part can be repaired or enhanced by using an image processing algorithm, so that the abnormal part is clearer and is easy to analyze and understand. Therefore, abnormal conditions can be better found and processed, and the driving safety of the vehicle is improved.
Further, the step of performing a sharpening process on the abnormal portion determined by the separation line according to the type of the abnormality factor includes:
identifying perspective clue data of the rearview space four-dimensional graph which is shielded by the anomaly factors;
and importing the perspective clue data into an image processing algorithm for element complementation, generating first complement data and loading the first complement data into an abnormal part of the four-dimensional graph of the rearview space.
This step is for the case where an anomaly factor exists in the rearview spatial four-dimensional map, where the anomaly factor may cause the separation line to be blocked, thereby affecting the sharpness and accuracy of the image. To solve this problem, a sharpening process is required, and the specific steps are as follows:
first, it is necessary to identify perspective cue data in the four-dimensional view of the rearview space that is occluded by the anomaly factors, which can be used to infer the portion that was originally occluded. This step may be accomplished by calculating the difference between adjacent regions. And then, the obtained perspective cue data is imported into an image processing algorithm to complete elements. When element complement is carried out, an image restoration algorithm based on a neural network can be used, and restoration of the missing part can be realized by learning structural information and texture features in the image. Thus, the first complement data can be generated. And finally, loading the first complement data into the abnormal part of the four-dimensional graph of the rearview space to fuse the first complement data with the normal part, thereby clearly processing the abnormal part and improving the quality and accuracy of the image.
In another embodiment, the step of performing the sharpening process on the abnormal portion determined by the separation line according to the type of the abnormality factor includes:
the deduction data of the rearview space four-dimensional graph which is completely blocked by the abnormal factors are identified;
and importing the deduction data into an image processing algorithm for element completion, generating second completion data and loading the second completion data into an abnormal part of the four-dimensional map of the rearview space. Firstly, deduction data which are completely blocked by an abnormal factor in a rearview space four-dimensional graph need to be identified. These data can be predicted and inferred using models and algorithms. For example, the motion state of the vehicle and information of adjacent areas can be used to infer the structure and texture features of the occluded part. Then, the obtained deduction data is imported into an image processing algorithm to complete elements. When element complement is carried out, an image restoration algorithm based on a neural network can be used, and restoration of the missing part can be realized by learning structural information and texture features in the image. This enables the generation of second complement data. And finally, loading the second complement data into the abnormal part of the four-dimensional graph of the rearview space to fuse the second complement data with the normal part, thereby clearly processing the abnormal part and improving the quality and accuracy of the image.
Referring to fig. 2, a block diagram of an image processing device based on a streaming rearview mirror according to the present invention includes:
the system comprises an acquisition unit 1, a display unit and a display unit, wherein the acquisition unit is used for acquiring a plurality of rearview mirror images through a streaming media rearview mirror, and the plurality of rearview mirror images at least comprise a left rearview mirror image and a right rearview mirror image;
the composition unit 2 is used for inputting a plurality of rearview mirror images into a four-dimensional streaming media modeling framework through corresponding data transmission channels respectively so as to generate a rearview space four-dimensional diagram through the four-dimensional streaming media modeling framework;
the calibration unit 3 is used for inputting the rear-view space four-dimensional graph into an image convolution neural network model so as to calibrate abnormal factors in the rear-view space four-dimensional graph through the image convolution neural network model;
and the processing unit 4 is used for performing sharpening processing on the rearview space four-dimensional graph calibrated with the abnormal factors through an image processing algorithm so as to generate a display image associated with the rearview space four-dimensional graph and display the display image.
In this embodiment, for specific implementation of each unit in the above embodiment of the apparatus, please refer to the description in the above embodiment of the method, and no further description is given here.
Referring to fig. 3, in an embodiment of the present invention, there is further provided a computer device, which may be a server, and an internal structure thereof may be as shown in fig. 3. The computer device includes a processor, a memory, a display screen, an input device, a network interface, and a database connected by a system bus. Wherein the computer is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used for communicating with an external terminal through a network connection. Which computer program, when being executed by a processor, carries out the above-mentioned method.
It will be appreciated by those skilled in the art that the architecture shown in fig. 3 is merely a block diagram of a portion of the architecture in connection with the present inventive arrangements and is not intended to limit the computer devices to which the present inventive arrangements are applicable.
An embodiment of the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the above method. It is understood that the computer readable storage medium in this embodiment may be a volatile readable storage medium or a nonvolatile readable storage medium.
In summary, a plurality of rearview mirror images are collected through the streaming media rearview mirror, wherein the plurality of rearview mirror images at least comprise a left rearview mirror image and a right rearview mirror image; inputting a plurality of rearview mirror images into a four-dimensional streaming media modeling framework through corresponding data transmission channels respectively so as to generate a rearview space four-dimensional diagram through the four-dimensional streaming media modeling framework; inputting the rearview space four-dimensional graph into an image convolution neural network model, and calibrating abnormal factors in the rearview space four-dimensional graph through the image convolution neural network model; performing sharpening processing on the rearview space four-dimensional map calibrated with the abnormal factors through an image processing algorithm to generate a display image associated with the rearview space four-dimensional map, and displaying the display image; the method comprises the steps of identifying the shielding objects such as raindrops and dust in the acquired image through a pre-trained image convolution model, marking the positions of the shielding objects, filling and repairing the marked shielding objects through an image processing algorithm of a layered channel, and achieving the effect of enabling the image to be restored to the optimal state.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium provided by the present invention and used in embodiments may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM, among others.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, apparatus, article or method that comprises the element.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the invention, and all equivalent structures or equivalent processes using the descriptions and drawings of the present invention or direct or indirect application in other related technical fields are included in the scope of the present invention.
Claims (9)
1. An image processing method based on a streaming media rearview mirror is characterized by comprising the following steps:
collecting a plurality of rearview mirror images through a streaming media rearview mirror, wherein the plurality of rearview mirror images at least comprise a left rearview mirror image and a right rearview mirror image;
inputting a plurality of rearview mirror images into a four-dimensional streaming media modeling framework through corresponding data transmission channels respectively so as to generate a rearview space four-dimensional diagram through the four-dimensional streaming media modeling framework;
inputting the rearview space four-dimensional graph into an image convolution neural network model, and calibrating abnormal factors in the rearview space four-dimensional graph through the image convolution neural network model;
performing sharpening processing on the rearview space four-dimensional map calibrated with the abnormal factors through an image processing algorithm to generate a display image associated with the rearview space four-dimensional map, and displaying the display image;
the method for creating the four-dimensional streaming media modeling framework comprises the following steps:
creating a vehicle-mounted streaming media coordinate system xyr, wherein the vehicle-mounted streaming media coordinate system xyr comprises a driving direction x, a rearview viewing angle y and a steering inclination angle r;
and establishing a data transmission protocol between the vehicle-mounted streaming media coordinate system xyr and a data transmission channel so as to correlate the marking information t pre-correlated with the data transmission channel with the rearview viewing angle y to create the four-dimensional streaming media modeling framework xyrt.
2. The method for processing images based on a streaming rearview mirror according to claim 1, wherein the step of inputting a plurality of rearview mirror images into a four-dimensional streaming modeling framework through corresponding data transmission channels to generate a rearview space four-dimensional map through the four-dimensional streaming modeling framework comprises:
setting a corresponding number of data transmission channels according to the specific number of the rearview mirror images;
identifying and marking the shooting directions of a plurality of rearview mirror images;
inputting a plurality of rearview mirror images marked with camera shooting directions into a four-dimensional streaming media modeling framework to generate a rearview space four-dimensional diagram by the four-dimensional streaming media modeling framework, wherein the generating method comprises the following steps of,
image stitching is carried out on a plurality of rearview mirror images according to the shooting direction marks;
identifying image edges of a plurality of rearview mirror images, and judging whether edge superposition occurs in image stitching of the plurality of rearview mirror images according to the image edges;
if the edges are overlapped, performing de-overlapping treatment after image splicing to generate a rearview space four-dimensional image;
and if the edges are not overlapped, carrying out broken line fitting processing after image splicing to generate a rearview space four-dimensional graph.
3. The image processing method based on the streaming media rearview mirror according to claim 1, wherein the method for calibrating the anomaly factors of the four-dimensional map of the rearview space by the image convolution neural network model is as follows:
continuously acquiring a rearview space four-dimensional map through an input layer, and importing the rearview space four-dimensional map into a convolution layer;
performing distance matrix segmentation processing on the rear-view space four-dimensional map through a convolution layer to perform matrix segmentation on front, middle and rear sections of the rear-view space four-dimensional map to generate a front section rear-view space map, a middle section rear-view space map and a rear section rear-view space map, and inputting the front section rear-view space map, the middle section rear-view space map and the rear section rear-view space map to a pooling layer;
the front section rearview space diagram, the middle section rearview space diagram and the rear section rearview space diagram are subjected to visual sharpening enhancement treatment through the pooling layer and output to the full-connection layer;
respectively matching abnormal factors in a front section rearview space diagram, a middle section rearview space diagram and a rear section rearview space diagram through a pre-trained factor template in the full-connection layer, and inputting the abnormal factors into the softmax layer, wherein the abnormal factors comprise, but are not limited to, water drops and dust;
the similarity judging process of the abnormal factors and the factor templates is carried out through the softmax layer, and when the similarity of the abnormal factors and the factor templates is judged to be larger than a set value, the abnormal factors are calibrated at the positions of the front section rearview space diagram, the middle section rearview space diagram and the rear section rearview space diagram;
and outputting the front section rearview space diagram, the middle section rearview space diagram and the rear section rearview space diagram calibrated with the abnormal factors by an output layer.
4. A streaming rearview mirror based image processing method according to claim 3, wherein the step of performing a sharpening process on the rearview space four-dimensional map calibrated with an abnormality factor by an image processing algorithm to generate a display image associated with the rearview space four-dimensional map and displaying the display image comprises:
identifying an abnormal part which is shielded by an abnormal factor and a normal part which is not shielded in the rearview space four-dimensional diagram;
sharpening the fuzzy line between the abnormal part and the normal part to generate a separation line;
and carrying out the sharpening process on the abnormal part determined by the separation line according to the type of the abnormal factor.
5. The method for processing an image based on a streaming rearview mirror according to claim 4, wherein the step of performing a sharpening process on the abnormal portion determined by the dividing line according to the type of the abnormality factor comprises:
identifying perspective clue data of the rearview space four-dimensional graph which is shielded by the anomaly factors;
and importing the perspective clue data into an image processing algorithm for element complementation, generating first complement data and loading the first complement data into an abnormal part of the four-dimensional graph of the rearview space.
6. The method for processing an image based on a streaming rearview mirror according to claim 4, wherein the step of performing a sharpening process on the abnormal portion determined by the dividing line according to the type of the abnormality factor comprises:
the deduction data of the rearview space four-dimensional graph which is completely blocked by the abnormal factors are identified;
and importing the deduction data into an image processing algorithm for element completion, generating second completion data and loading the second completion data into an abnormal part of the four-dimensional map of the rearview space.
7. An image processing apparatus based on a streaming rearview mirror, comprising:
the system comprises an acquisition unit, a control unit and a control unit, wherein the acquisition unit is used for acquiring a plurality of rearview mirror images through a streaming media rearview mirror, and the plurality of rearview mirror images at least comprise a left rearview mirror image and a right rearview mirror image;
the composition unit is used for inputting a plurality of rearview mirror images into a four-dimensional streaming media modeling framework through corresponding data transmission channels respectively so as to generate a rearview space four-dimensional diagram through the four-dimensional streaming media modeling framework;
the calibration unit is used for inputting the rearview space four-dimensional graph into an image convolution neural network model so as to calibrate abnormal factors in the rearview space four-dimensional graph through the image convolution neural network model;
the processing unit is used for carrying out sharpening processing on the rearview space four-dimensional graph calibrated with the abnormal factors through an image processing algorithm so as to generate a display image associated with the rearview space four-dimensional graph and display the display image;
the method for creating the four-dimensional streaming media modeling framework comprises the following steps:
creating a vehicle-mounted streaming media coordinate system xyr, wherein the vehicle-mounted streaming media coordinate system xyr comprises a driving direction x, a rearview viewing angle y and a steering inclination angle r;
and establishing a data transmission protocol between the vehicle-mounted streaming media coordinate system xyr and a data transmission channel so as to correlate the marking information t pre-correlated with the data transmission channel with the rearview viewing angle y to create the four-dimensional streaming media modeling framework xyrt.
8. A computer device comprising a memory and a processor, the memory having stored therein a computer program, characterized in that the processor, when executing the computer program, carries out the steps of the streaming mirror based image processing method according to any of claims 1 to 6.
9. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the streaming mirror based image processing method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311462899.4A CN117237393B (en) | 2023-11-06 | 2023-11-06 | Image processing method and device based on streaming media rearview mirror and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311462899.4A CN117237393B (en) | 2023-11-06 | 2023-11-06 | Image processing method and device based on streaming media rearview mirror and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117237393A true CN117237393A (en) | 2023-12-15 |
| CN117237393B CN117237393B (en) | 2024-05-17 |
Family
ID=89091503
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311462899.4A Active CN117237393B (en) | 2023-11-06 | 2023-11-06 | Image processing method and device based on streaming media rearview mirror and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117237393B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE102015011831A1 (en) * | 2015-09-09 | 2017-03-09 | Michael Wenzel | Motor vehicle exterior rearview mirror with camera-monitor system for indirect visual representations of the oncoming lane and a vehicle in front in the direction of travel |
| US20190379841A1 (en) * | 2017-02-16 | 2019-12-12 | Jaguar Land Rover Limited | Apparatus and method for displaying information |
| CN114821531A (en) * | 2022-04-25 | 2022-07-29 | 广州优创电子有限公司 | Lane line recognition image display system based on electronic outside rear-view mirror ADAS |
| CN115114963A (en) * | 2021-09-24 | 2022-09-27 | 中国劳动关系学院 | Intelligent streaming media video big data analysis method based on convolutional neural network |
| CN115195605A (en) * | 2022-08-11 | 2022-10-18 | 广州小鹏自动驾驶科技有限公司 | Data processing method and device based on streaming media rearview mirror system and vehicle |
| WO2022266101A1 (en) * | 2021-06-14 | 2022-12-22 | The Johns Hopkins University | Systems, methods, and computer programs for using a network of machine learning models to detect an image depicting an object of interest which can be partially occluded by another object |
| US20230026706A1 (en) * | 2021-07-23 | 2023-01-26 | International Business Machines Corporation | Selective mirror enhanced video stream |
| US20230044180A1 (en) * | 2020-01-17 | 2023-02-09 | Semiconductor Energy Laboratory Co., Ltd. | Imaging system, driving assistance system, and program |
| CN115965531A (en) * | 2022-12-28 | 2023-04-14 | 华人运通(上海)自动驾驶科技有限公司 | Model training method, image generation method, device, equipment and storage medium |
| CN116205829A (en) * | 2022-12-30 | 2023-06-02 | 武汉市航盛汽车电子有限公司 | Rearview mirror image fusion method, device, vehicle-mounted equipment and storage medium |
| US20230252793A1 (en) * | 2022-02-08 | 2023-08-10 | Continental Autonomous Mobility Germany GmbH | Rear-view stereo camera system |
-
2023
- 2023-11-06 CN CN202311462899.4A patent/CN117237393B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE102015011831A1 (en) * | 2015-09-09 | 2017-03-09 | Michael Wenzel | Motor vehicle exterior rearview mirror with camera-monitor system for indirect visual representations of the oncoming lane and a vehicle in front in the direction of travel |
| US20190379841A1 (en) * | 2017-02-16 | 2019-12-12 | Jaguar Land Rover Limited | Apparatus and method for displaying information |
| US20230044180A1 (en) * | 2020-01-17 | 2023-02-09 | Semiconductor Energy Laboratory Co., Ltd. | Imaging system, driving assistance system, and program |
| WO2022266101A1 (en) * | 2021-06-14 | 2022-12-22 | The Johns Hopkins University | Systems, methods, and computer programs for using a network of machine learning models to detect an image depicting an object of interest which can be partially occluded by another object |
| US20230026706A1 (en) * | 2021-07-23 | 2023-01-26 | International Business Machines Corporation | Selective mirror enhanced video stream |
| CN115114963A (en) * | 2021-09-24 | 2022-09-27 | 中国劳动关系学院 | Intelligent streaming media video big data analysis method based on convolutional neural network |
| US20230252793A1 (en) * | 2022-02-08 | 2023-08-10 | Continental Autonomous Mobility Germany GmbH | Rear-view stereo camera system |
| CN114821531A (en) * | 2022-04-25 | 2022-07-29 | 广州优创电子有限公司 | Lane line recognition image display system based on electronic outside rear-view mirror ADAS |
| CN115195605A (en) * | 2022-08-11 | 2022-10-18 | 广州小鹏自动驾驶科技有限公司 | Data processing method and device based on streaming media rearview mirror system and vehicle |
| CN115965531A (en) * | 2022-12-28 | 2023-04-14 | 华人运通(上海)自动驾驶科技有限公司 | Model training method, image generation method, device, equipment and storage medium |
| CN116205829A (en) * | 2022-12-30 | 2023-06-02 | 武汉市航盛汽车电子有限公司 | Rearview mirror image fusion method, device, vehicle-mounted equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| FITRI RAHMAH等: "Radial line method for rear-view mirror distortion detection", INTERNATIONAL SEMINAR ON PHOTONICS, OPTICS, AND ITS APPLICATIONS, 31 December 2015 (2015-12-31), pages 1 - 5 * |
| 刘志强;王天;冯新颖;: "后视镜可视化在汽车驾驶仿真系统中的实现", 计算机仿真, no. 02, 15 February 2015 (2015-02-15), pages 173 - 177 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117237393B (en) | 2024-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9773176B2 (en) | Image processing apparatus and image processing method | |
| DE102008053472B4 (en) | Object detection system | |
| DE102013223039B4 (en) | Apparatus and method for providing an AVM image | |
| CN105763854B (en) | A kind of omnidirectional imaging system and its imaging method based on monocular cam | |
| KR101811157B1 (en) | Bowl-shaped imaging system | |
| CN110706282A (en) | Automatic calibration method and device for panoramic system, readable storage medium and electronic equipment | |
| CN111179168B (en) | Vehicle-mounted 360-degree panoramic all-around monitoring system and method | |
| EP3671643A1 (en) | Method and apparatus for calibrating the extrinsic parameter of an image sensor | |
| DE102013220669A1 (en) | Dynamic rearview indicator features | |
| CN109741241B (en) | Fisheye image processing method, device, equipment and storage medium | |
| DE102013226476B4 (en) | IMAGE PROCESSING METHOD AND SYSTEM OF AN ALL-ROUND SURVEILLANCE SYSTEM | |
| CN112224132A (en) | Vehicle panoramic all-around obstacle early warning method | |
| CN110719411B (en) | Panoramic all-around view image generation method of vehicle and related equipment | |
| CN110378836B (en) | Method, system and equipment for acquiring 3D information of object | |
| CN111768332A (en) | Splicing method of vehicle-mounted all-around real-time 3D panoramic image and image acquisition device | |
| CN114372919B (en) | Method and system for splicing panoramic all-around images of double-trailer train | |
| DE102021124986A1 (en) | IMAGE COLORING FOR VEHICLE CAMERA IMAGES | |
| CN110796711A (en) | Panoramic system calibration method and device, computer readable storage medium and vehicle | |
| EP3326146B1 (en) | Rear cross traffic - quick looks | |
| CN111160070A (en) | Vehicle panoramic image blind area eliminating method and device, storage medium and terminal equipment | |
| CN112825546A (en) | Generating a composite image using an intermediate image surface | |
| DE112016007407T5 (en) | DISPLAY CONTROL DEVICE, DISPLAY SYSTEM AND DISPLAY CONTROL METHOD | |
| US11377027B2 (en) | Image processing apparatus, imaging apparatus, driving assistance apparatus, mobile body, and image processing method | |
| CN117237393B (en) | Image processing method and device based on streaming media rearview mirror and computer equipment | |
| US20220408062A1 (en) | Display control apparatus, display control method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |