CN115297230B - System and method for multiplexing electronic exterior rearview mirror and intelligent driving side rearview camera - Google Patents
System and method for multiplexing electronic exterior rearview mirror and intelligent driving side rearview camera Download PDFInfo
- Publication number
- CN115297230B CN115297230B CN202210767172.6A CN202210767172A CN115297230B CN 115297230 B CN115297230 B CN 115297230B CN 202210767172 A CN202210767172 A CN 202210767172A CN 115297230 B CN115297230 B CN 115297230B
- Authority
- CN
- China
- Prior art keywords
- image data
- camera
- image
- multiplexing
- path
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000012545 processing Methods 0.000 claims abstract description 84
- 238000000605 extraction Methods 0.000 claims abstract description 26
- 230000008569 process Effects 0.000 claims abstract description 15
- 238000012937 correction Methods 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000006835 compression Effects 0.000 description 6
- 238000007906 compression Methods 0.000 description 6
- 238000001514 detection method Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 230000000007 visual effect Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000003709 image segmentation Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 2
- 101100248200 Arabidopsis thaliana RGGB gene Proteins 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
- B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
- B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
- B60R1/23—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view
- B60R1/25—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view to the sides of the vehicle
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
- B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
- B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
- B60R1/23—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view
- B60R1/26—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view to the rear of the vehicle
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Mechanical Engineering (AREA)
- Studio Devices (AREA)
Abstract
A system and a method for multiplexing an electronic exterior rearview mirror and an intelligent driving side rearview camera are provided, wherein the system comprises a multiplexing camera, and an electronic exterior rearview mirror processing module and a domain controller which are respectively connected with the multiplexing camera. The electronic exterior rearview mirror processing module processes the first path of image data and outputs exterior rearview mirror images, the second path of image data is transmitted to the domain controller, and the domain controller performs frame extraction processing on the second path of image data, so that the frame-extracted image data accords with the condition of a side rear view image recognition algorithm. According to the invention, the electronic outside rearview mirror system and the intelligent driving domain side rearview system share the mode of one camera through the two systems, and the side rearview camera is canceled, so that the complexity of a hardware structure is reduced, the contradiction of hardware arrangement is relieved, and the hardware conflict is reduced.
Description
Technical Field
The invention relates to a vehicle intelligent detection system and method, in particular to a system and method for multiplexing an electronic outside rearview mirror and a intelligent driving side rearview camera.
Background
Electronic outside mirrors are a hot spot direction of current vehicle development, and although existing vehicles are rarely equipped, there is a trend in the future. The electronic outside rearview mirror forms images through the camera, and rear view images are projected onto the display screen in the automobile, so that the traditional lens type rearview mirror is replaced.
In addition to electronic exterior rear view mirrors, there are already a variety of cameras on existing vehicles, such as side rearview systems for intelligent driving. The intelligent driving side rearview system is an intelligent driving auxiliary system based on a visual technology, and mainly realizes side rear coming vehicles and object monitoring by utilizing a domain controller to collect data and identify images of a plurality of cameras of the whole vehicle, and achieves lane changing auxiliary and early warning functions. The electronic outside rear-view mirror system hardware is composed of a high-pixel camera, an image processor, a controller and a display screen, provides a more flexible visual angle and wider visual coverage for a driver, and improves the visibility in dark, rainy days and other environments with poor light under the holding of the camera technology.
The exterior of the existing vehicle type on the market is provided with 11 or more intelligent driving cameras, which generally comprise two front-view cameras, two side rear-view cameras, one rear-view camera and four round-view cameras, if an electronic exterior rearview mirror system is adopted, two rear-view cameras are additionally added at the rearview mirror, and at least 13 cameras are assembled outside the vehicle.
Therefore, along with the continuous improvement of the whole vehicle technology, the number of cameras of the vehicle is increased, so that the problems of overall hardware cost, hardware conflict, structural complexity and the like are also more and more prominent.
Disclosure of Invention
Aiming at the problems of increased cameras of the whole vehicle, complex structure and the like in the prior art, the invention provides a system and a method for multiplexing an electronic outside rearview mirror and a rearview camera at the intelligent driving side, which at least can multiplex the electronic outside rearview mirror and the rearview camera at the intelligent driving side, so as to reduce the complexity of a hardware structure and hardware conflict.
In order to achieve the above purpose, the invention adopts the following technical scheme:
A system for multiplexing an electronic exterior rearview mirror and an intelligent driving side rearview camera comprises a multiplexing camera, and an electronic exterior rearview mirror processing module and a domain controller which are respectively connected with the multiplexing camera. The electronic exterior rearview mirror processing module processes the first path of image data and outputs exterior rearview mirror images, the second path of image data is transmitted to the domain controller, and the domain controller performs frame extraction processing on the second path of image data, so that the frame-extracted image data accords with the condition of a side rear view image recognition algorithm.
As one embodiment of the invention, the electronic outside rear view mirror processing module comprises a first deserializer, a first serializer, a first image processor and a controller, wherein the first deserializer is respectively connected with the first serializer and the first image processor, and the first image processor is further connected with the controller. The first deserializer receives serial image data of the multiplexing camera, deserializes the serial image data, divides the serial image data into two identical paths, and transmits the two paths to the first serializer and the first image processor respectively. The first image processor processes the first path of image data and then transmits the processed image data to the controller, and the controller processes and outputs the outside rearview mirror image.
As an embodiment of the present invention, the domain controller includes a second deserializer and a second image processor, wherein the second deserializer is connected to the first serializer and the second image processor, respectively. The first serializer transmits the second path of image data to the second deserializer in series, and the second deserializer carries out frame extraction processing on the second path of image data, so that the image data after frame extraction accords with the condition of a side rear view image recognition algorithm.
As one embodiment of the present invention, the multiplexed camera includes an image sensor that acquires image data and transmits the image data to a second serializer that serially transmits the image to a first deserializer.
As an embodiment of the present invention, the domain controller controls the multiplexed camera to be exposed simultaneously with other vehicle-mounted cameras, and the electronic outside rear view mirror processing module controls the exposure time length and white balance of the multiplexed camera.
As one embodiment of the present invention, the same two-way image data are image data of at least 60 frames in frame rate, and the domain controller decimates the second-way image data into image data of 30 frames in frame rate.
In order to achieve the above purpose, the present invention further adopts the following technical scheme:
a multiplexing method of an electronic exterior rearview mirror and a intelligent driving side rearview camera comprises the following steps: acquiring vehicle body side rear image data; dividing the image data into two identical paths; processing the first path of image data and outputting an outer rearview mirror image; and performing frame extraction processing on the second path of image data, so that the image data subjected to frame extraction accords with the condition of a side rear view image recognition algorithm.
As one embodiment of the present invention, the electronic outside mirror processing module is utilized to decompress the first path of image data, and at least one of the following image processing operations is performed: black level compensation, bad pixel correction, color interpolation, denoising, white balance correction, color correction, gamma correction, color space conversion, color noise removal, and edge enhancement. Outputting the processed first path of image data to an outside rearview mirror image; and serially transmitting the second path of image data to the domain controller.
As one embodiment of the present invention, the second path of image data is deserialized by using a domain controller; and performing frame extraction processing on the second path of image data, so that the image data subjected to frame extraction accords with the condition of a side rear view image recognition algorithm.
As one embodiment of the invention, the domain controller is used for controlling the multiplexing camera and other vehicle-mounted cameras to be exposed simultaneously so as to obtain the image data of the rear side of the vehicle body; and controlling the exposure time and white balance of the multiplexing camera by using the electronic outside rearview mirror processing module.
As one embodiment of the present invention, the same two-way image data are image data of at least 60 frames in frame rate, and the domain controller decimates the second-way image data into image data of 30 frames in frame rate.
In the technical scheme, the electronic outside rearview mirror system and the intelligent driving domain side rearview system share the mode of one camera through the two systems, and the side rearview camera is canceled, so that the complexity of a hardware structure is reduced, the contradiction of hardware arrangement is relieved, and the hardware conflict is reduced.
Drawings
FIG. 1a is a schematic view of an architecture of an electronic exterior mirror selected in accordance with the present invention;
FIG. 1b is a schematic diagram of an architecture of a selected domain controller of the present invention;
FIG. 2 is a schematic diagram of the overall architecture of the system of the present invention;
Fig. 3 is a flow chart of the method of the present invention.
In the figure:
10-cameras, 11-image sensors, 12-serializers;
20-an electronic outside rear-view mirror processing module, 21-a deserializer, 22-an image processor and 23-a controller;
30-domain controller, 31-deserializer, 32-image processor;
40-camera, 41-image sensor, 42-serializer;
50-multiplexing camera, 51-image sensor, 52-second serializer;
60-an electronic exterior rear view mirror processing module, 61-a first deserializer, 62-a first image processor, 63-a first serializer, 64-a controller;
70-domain controller, 71-second deserializer, 72-second image processor.
Detailed Description
The technical solutions in the embodiments of the present invention are further clearly and completely described below with reference to the accompanying drawings and the embodiments. It is clear that the examples described are for the purpose of explaining the technical solution of the invention and are not meant to be exhaustive of all embodiments of the invention.
Examples of the embodiments are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements throughout or elements having like or similar functionality. The embodiments described below by referring to the drawings are illustrative and intended to explain the present invention and should not be construed as limiting the invention.
Because the arrangement positions of the electronic exterior rearview mirror and the side rearview camera are very close to each other in the vehicle body, and the functions realized by the electronic exterior rearview mirror and the side rearview camera are similar, the invention considers whether the close-view camera, namely the electronic exterior rearview mirror and the side rearview camera can be reused or not. However, there are various implementations of the electronic outside rear view mirror and the side rearview camera, and if multiplexing of the two is to be achieved, it is first necessary to select an electronic outside rear view mirror system and an intelligent driving side rearview system that are relatively close in architecture. In addition, since the electronic outside rear view mirror system and the intelligent driving side rear view system both have ready-made software algorithms, if multiplexing of the two systems is to be realized, the software algorithms of the two systems should be kept as unchanged (or as unchanged) as possible, otherwise, the algorithms of the two systems should be redesigned, so that the system is not required to be damaged.
Based on the above requirements, the present invention first selects an electronic outside rear view mirror system (CMS, camera Monitor System) and an intelligent driving side rear view system (IPD, INTELLIGENT PILOT DEVICE) of a specific architecture, as shown in fig. 1a and 1b, which are close to each other in overall architecture, and both are relatively simple in structure, so that multiplexing of both can be facilitated.
Fig. 1a is a schematic view showing the structure of an electronic exterior mirror according to an embodiment of the present invention. Referring to fig. 1a, the electronic exterior mirror system selected in the present invention mainly includes a camera 10 and an electronic exterior mirror processing module 20. The camera 10 further includes an image sensor 11 and a serializer 12, and the electronic outside rear view mirror processing module 20 mainly includes a deserializer 21, an image processor 22, and a controller 23.
In the system shown in fig. 1a, the image sensor 11 acquires an image and transmits the image to the serializer 12, and the serializer 12 forms the image data into a serial format and serially transmits the image data to the deserializer 21. The deserializer 21 deserializes the serial image data and sends it to the image processing unit 22, which performs normalization processing on the image data and transfers the processed image data to the controller 23. The controller 23 displays the image data on a display screen in the vehicle to form an electronic outside mirror image.
Fig. 1b is a schematic diagram of an architecture of an intelligent driving side rearview system (part of a intelligent driving system) selected by the present invention. Referring to fig. 1b, the intelligent driving side rearview system selected in the present invention mainly includes a camera 40 and a domain controller 30. The camera 40 further includes an image sensor 41 and a serializer 42, and the domain controller 30 mainly includes a deserializer 31 and an image processor 32.
In the system shown in fig. 1b, the image sensor 41 acquires an image and transmits the image to the serializer 42, and the serializer 42 forms the image data into a serial format and serially transmits the image data to the deserializer 31. The deserializer 31 deserializes the serial image data and sends the image data to the image processor 32, the image processing unit performs standardization processing on the image data, and combines the processed image data with other parameters in the intelligent driving side rearview system so as to provide intelligent driving assistance, such as object monitoring, lane changing assistance, early warning functions and the like.
Based on the electronic outside rearview mirror system and the intelligent driving side rearview system shown in fig. 1a and 1b, the invention multiplexes cameras of the two systems and fine-adjusts the system structure to form the system architecture shown in fig. 2. The core point of the camera multiplexing of the invention is that: the side rearview camera and the electronic exterior rearview mirror camera have similar visual fields, the two systems share the same camera hardware, a raw image output by the camera is differentially transmitted from a transmission link of the electronic exterior rearview mirror to the side rearview system, the raw image is transmitted to the intelligent driving domain controller for frame extraction, and then image processing is carried out, so that machine identification is realized, and vehicle control is completed.
Referring to fig. 2, the multiplexing system of the present invention mainly includes a multiplexing camera 50, and electronic outside rear view mirror processing modules 60 (CMS, camera Monitor System) respectively connected to the multiplexing camera 50, and a domain controller 70 for an intelligent driving side rear view system (IPD, INTELLIGENT PILOT DEVICE). The multiplexed camera 50 mainly includes an image sensor 51 and a second serializer 52, the electronic outside rear view mirror Processing module 60 mainly includes a first deserializer 61, a first serializer 63, a first image processor 62 (ISP, image Signal Processing), and a controller 64, and the domain controller 70 mainly includes a second deserializer 71 and a second image processor 72 (ISP, image Signal Processing).
As can be seen from the structure shown in fig. 2, the present invention proposes a system and a method for enabling an electronic exterior rearview mirror processing module and a side rearview intelligent driving system to share the same camera sensor, so as to achieve the purpose of reducing one camera. The multiplexing system maintains the structures of the original electronic outside rearview mirror system and the intelligent driving side rearview system unchanged to the greatest extent, and only adds the first serializer on the basis of the original electronic outside rearview mirror system, and in addition, the structures of the multiplexing camera and the original camera are basically consistent.
The invention maintains the unchanged framework of the camera, the electronic outside rearview mirror system and the intelligent driving side rearview system, and has the following significance: because the system architecture has no fundamental change, the software algorithms of the original camera, the electronic outside rearview mirror system and the intelligent driving side rearview system can be reserved and used to the greatest extent, and a new multiplexing algorithm does not need to be developed by pushing over. The invention can realize the system multiplexing of hardware and software layers by only adding a specific algorithm corresponding to the multiplexing system on the basis of the original algorithm.
Referring to fig. 3, the invention also discloses a method for multiplexing the electronic outside rearview mirror and the intelligent driving side rearview camera.
The method has the core ideas that: the image sensor of the multiplexing camera 50 is controlled by a domain controller 70 to start exposure through hard wires, the first image processor 62 of the electronic outside rear view mirror processing module 60 controls exposure time and white balance, a serial deserializer is used for deserializing after generating a picture, one path of the deserialized image is transmitted to the first image processor 62 of the electronic outside rear view mirror processing module 60 for processing, and then the image is subjected to picture adjustment and the like through the controller 64 of the electronic outside rear view mirror processing module 60 and finally is transmitted to a large screen to be displayed as a rear view mirror image for a driver to watch. The other path of transmission is transmitted to a second image processor 72 in the domain controller 70 through serial deserialization, is fused with other intelligent driving system cameras, completes the monitoring of the object at the rear side through the machine identification of an intelligent driving algorithm, and sends an instruction to the whole vehicle to complete lane change early warning and lane change auxiliary functions.
The multiplexing method of the present invention is described below in conjunction with the system architecture of fig. 2.
With continued reference to fig. 2, a multiplexed camera 50 is provided on the vehicle body side, which acquires image data by an image sensor 51 and transmits the image data to a second serializer 52, and the second serializer 52 serially transmits the image to a first deserializer 61.
As can be seen from the above architecture, in connection with fig. 2 and 3, the multiplexing camera 50 first performs step 1: and acquiring vehicle body side rear image data. Step 1 is realized by the following sub-steps:
s1: the multiplexed camera 50 is controlled by the domain controller 70 to be exposed simultaneously with other in-vehicle cameras, thereby obtaining vehicle-body-side rear image data.
Specifically, the domain controller 70 simultaneously exposes the multiplexed camera 50 (serving as a side-rearview camera) and other in-vehicle cameras through hard-wire control, and the image processor in the electronic outside-mirror processing module 60 controls the exposure time period according to the outside environment brightness, and performs white balance processing. At this time, the electronic outside mirror processing module 60 and the domain controller 70 need to be pulled up in time sequence, that is, the exposure time of the multiplexed camera 50 is set to the start time of one frame, so as not to read the exposure when the exposure is not finished yet.
The domain controller 70 controls the exposure of the multiplexed camera 50 by hard wire, the exposure duration being controlled by the first image processor 62 in the electronic outside mirror processing module 60. When the multiplexing camera 50 is used as a side rearview camera, the multiplexing camera belongs to a intelligent driving camera system, and needs to be visually fused with other vehicle-mounted cameras, so that the domain controller 70 needs to control the multiplexing camera 50 and other intelligent driving cameras to be exposed at the same time, and the time deviation of each camera data reaching an image processor needs to be within 1ms so as to ensure that all cameras acquire images at the same time.
S2: the electronic exterior mirror processing module 60 is utilized to control the exposure time and white balance of the multiplexed camera 50.
After the multiplexed camera 50 starts exposure, the electronic outside rear view mirror processing module 60 runs the built-in firmware through the first image processor 62 inside the electronic outside rear view mirror processing module to correspondingly control the image sensor 51 of the multiplexed camera 50, complete automatic exposure, record exposure duration and automatic white balance.
Since the human eye recognition requires higher image quality than machine recognition, when the multiplexed camera 50 is used as the electronic outside rear view mirror processing module 60, it uses an adaptive exposure algorithm, so the present invention adaptively adjusts the exposure time period according to the ambient light by the second image processor 72 in the electronic outside rear view mirror processing module 60.
S3: the image sensor 51 of the multiplexed camera 50 generates a raw format image (raw/bayer image) and compresses it according to an image segmentation compression curve (PWL). The image sensor 51 then transmits the compressed image data to the second serializer 52, and the second serializer 52 serializes the image data and then transmits the serial image data to the electronic outside rear view mirror processing module 60.
In conjunction with the above system architecture and method, as a first problem to be solved by the multiplexing system and method, the present invention needs to meet the time synchronization requirement of each camera of the intelligent driving system, that is, the multiplexing camera 50 of the present invention needs to keep time synchronization with the cameras of other intelligent driving systems connected to the domain controller 70. Such a problem is not present when the electronic outside mirror system, the intelligent driving side rearview system are independent of each other, as shown in the architecture of fig. 1a and 1b, the cameras of both can be operated separately and without synchronization requirements, and this problem must be solved after multiplexing the electronic outside mirror system, the intelligent driving side rearview system. Therefore, steps S1 and S2 of the present invention control the multiplexed camera 50 to be simultaneously exposed with other vehicle-mounted cameras by using the domain controller 70, and control the exposure time period and white balance of the multiplexed camera 50 by using the electronic outside mirror processing module 60.
As further shown in fig. 2, the electronic exterior mirror processing module 60 receives the serial image data of the multiplexed camera 50, divides the serial image data into two identical paths, processes the first path of image data, outputs the processed image data as an exterior mirror image, and transmits the second path of image data to the domain controller 70. Specifically, inside the electronic outside-mirror processing module 60, the first deserializer 61 is connected to the first serializer 63 and the first image processor 62, respectively, and the first image processor 62 is further connected to the controller 64. The first deserializer 61 receives the serial image data of the multiplexed camera 50, deserializes the serial image data, divides the same into two paths, and transmits the same to the first serializer 63 and the first image processor 62, respectively. The first image processor 62 processes the first path of image data and transmits the processed image data to the controller 64, and the controller 64 processes and outputs the exterior mirror image.
As can be seen from the above architecture, with reference to fig. 2 and 3, the electronic outside mirror processing module 60 mainly performs step 2: dividing the image data into two identical paths, and processing the first path of image data to output an outside rearview mirror image. Step 2 is realized by the following sub-steps:
s4: the serial, compressed image data of the multiplexed camera 50 is deserialized by the first deserializer 61.
S5: the first deserializer 61 divides the deserialized compressed image data into the same two paths.
S6: the first deserializer 61 transmits the first path of image data to the first image processor 62.
S7: the first image processor 62 decompresses the first path of image data according to the image segmentation compression curve.
S8: the first image processor 62 performs at least one or a combination of the following image processing operations according to the requirements of the back-end controller 64: black level compensation, bad pixel correction, color interpolation, denoising, white balance correction, color correction, gamma correction, color space conversion, color noise removal, and edge enhancement.
S9: the first image processor 62 transmits the processed first image data to the controller 64 at the rear end through the IO interface, and the controller 64 further outputs the first image data as an exterior mirror image on the in-vehicle display.
S10: for the second path of image data, the electronic outside mirror processing module 60 serially transmits the second path of image data to the domain controller 70 using the first serializer 63.
S11: the second path of image data (compressed data) is serially transmitted by the first serializer 63.
As can be seen from the above system architecture and method, one key step of the method of the present invention is the step S5 described above, which divides the decompressed compressed image data into two identical paths, namely, the first path image data and the second path image data. The same two paths of image data help to minimize changes in the internal structure, transmission relationships of the electronic exterior mirror processing module 60 and the domain controller 70, thereby reducing overall multiplexing system structure changes and further reducing algorithmic changes.
Continuing to refer to fig. 2, the domain controller 70 performs frame extraction processing on the second path of image data, so that the image data after frame extraction meets the condition of the side rear view image recognition algorithm. Specifically, inside the domain controller 70, the second deserializer 71 connects the first serializer 63 and the second image processor 72, respectively. The first serializer 63 serially transmits the second path of image data to the second deserializer 71, and the second deserializer 71 performs frame extraction processing on the second path of image data so that the image data after frame extraction meets the condition of the side rear view image recognition algorithm.
As can be seen from the above architecture, in connection with fig. 2 and 3, the domain controller 70 mainly performs step 3: and performing frame extraction processing on the second path of image data, so that the image data subjected to frame extraction accords with the condition of a side rear view image recognition algorithm. Step 3 is realized by the following sub-steps:
S12: the second image data transmitted from the first serializer 63 is deserialized by the second deserializer 71.
S13: the second deserializer 71 transmits the second path of image data after the deserialization to the second image processor 72.
S14: the second image processor 72 performs frame extraction processing on the second path image, so that the image data after frame extraction accords with the condition of the side rear view image recognition algorithm, and the second path image data is convenient for machine recognition of a later intelligent driving system.
In conjunction with the above system architecture and method, as a second problem to be solved by the multiplexing system and method, the present invention needs to solve the requirements of the electronic outside mirror processing module 60 and the domain controller 70 for the frame rate, respectively. The present invention processes the same two-way image data into image data of at least 60 frames in frame rate, and domain controller 70 processes the second-way image data into image data of 30 frames in frame rate.
The above settings are not arbitrary choices, but are dependent on the system parameters of the electronic outside mirror processing module 60 and the domain controller 70, respectively, so that the original image processing algorithms of the electronic outside mirror processing module 60 and the domain controller 70 are not changed as much as possible.
Specifically, the image frequency provided by the camera of the domain controller 70 (intelligent driving system) is 30 frames, that is, 30 pictures are sent out per second to machine recognition, and the electronic exterior mirror processing module 60 needs to achieve 60 frames for human eye recognition, so that the image frequency/frame rate can meet the requirement that the human eye observation is not stuck, and when the second path of image data generated by the electronic exterior mirror processing module 60 is used for machine recognition of the intelligent driving system, frame extraction algorithm processing is needed, that is, the original 60 frame frequency image is extracted to 30 frames for visual fusion with other cameras.
Those skilled in the art will appreciate that the multiplexing system architecture presented in fig. 2 is only one of many embodiments of the present invention and is not a limitation of the present invention. In the architecture shown in fig. 2, the multiplexed camera 50, the electronic outside mirror processing module 60, and the domain controller 70 embody their logical relationships with each other, and the logical relationships inside the respective modules, not the complete hardware relationships. In other embodiments of the present invention, the algorithm of the electronic exterior mirror processing module 60 may be integrated in the intelligent driving domain controller 70, and the same camera may be multiplexed or different multiplexing camera 50 parameters may be selected, so as to achieve the technical purpose of the present invention and achieve the technical effect of the present invention.
As an embodiment of the present invention, the sensor of the multiplexing camera 50 needs to meet the requirements of the electronic exterior rear view mirror processing module 60 and the domain controller 70 for the sensor, including the field of view after loading, the color filter matrix, the resolution, the CMOS chip, the frame rate, the dynamic range, the signal to noise ratio, the working temperature, the focal length error in the AA assembled finished product, the main point and image center error, and the internal reference calibration and internal reference detection. From the viewpoint of camera hardware parameters, the system requirements of the electronic exterior mirror processing module 60 are higher, so the invention selects to adapt to the original requirements of the electronic exterior mirror processing module 60. Because the intelligent driving algorithm is fused, higher assembly and internal reference calibration and detection accuracy are required, and the assembly and internal reference calibration and detection standard refers to the side rearview camera system.
As another embodiment of the present invention, the electronic outside rear-view mirror processing module 60 mainly displays the output image of the multiplexing camera 50 on the screen in the cabin, and the driver judges and recognizes that the physical coordinate system in the actual environment of the object in the acquired image is not required to be high, so that the internal reference calibration is not generally performed. However, for the domain controller 70, it is necessary to acquire the position of the object coordinate point in the real environment, so as to control the behavior of the whole vehicle, so that the accuracy of converting the pixel point coordinate captured based on the coordinate system of the multiplexing camera 50 into the real world coordinate system is required to be high, and the conversion accuracy is determined by the calibration error of the internal parameters of the camera. Thus, to ensure that the domain controller 70 obtains a more accurate coordinate system, higher internal reference calibration and detection error requirements are required when multiplexing the camera 50 sensor out of the field.
The generated map controlled by the first image processor 62 of the electronic outside mirror processing module 60 may be received and processed by the second image processor 72 in the domain controller 70. Different camera sensors are provided with segmented compression curves when coming out, and different image processor manufacturers adjust curve nodes for image processing adaptation. If the generated image controlled by the first image processor 62 of the electronic outside mirror processing module 60 is received and decompressed by the second image processor 72 in the domain controller 70, then both image processors are required to use the same compression curve.
According to the requirements, the CMOS of the multiplexing camera is IMX623, and the multiplexing system of the invention needs to modify an image segmentation compression curve (PWL) of IMX623 factory according to the requirements into a shared PWL curve which can be used by both an electronic outside rearview mirror system and an intelligent driving side rearview system, so that the compression curve controlled by an image processor of an electronic outside rearview mirror processing module can be decompressed by a domain controller.
In order to meet the requirements of the electronic exterior rearview mirror system and the intelligent driving side rearview mirror system on the camera, the invention selects a field of view (FOV) as follows: h72, V34, resolution: 1920×1536, a frame rate of 60 frames, and a color filter matrix (CFA, color FILTER ARRAY) of RGGB camera hardware as a multiplexing camera. The parameters and assembly of the specific cameras, and the calibration and detection errors of the internal parameters are shown in the following table 1:
Table 1: and selecting the parameters of the camera.
The apparatus of the invention may be a computer program product, wherein the computer program product when run on a computer causes the computer to perform some or all of the steps of the method as in the method embodiments above.
The apparatus of the invention may be an application publishing platform for publishing a computer program product, wherein the computer program product when run on a computer causes the computer to perform some or all of the steps of the method as in the method embodiments above.
In various embodiments of the present invention, it should be understood that the sequence numbers of the foregoing processes do not imply that the execution sequences of the processes should be determined by the functions and internal logic of the processes, and should not be construed as limiting the implementation of the embodiments of the present invention.
The various systems, units of the invention, if implemented in the form of software functional units, may be stored in a computer-accessible memory. With such understanding, some or all of the aspects of the present invention may be embodied in the form of a software product stored in a memory, comprising a number of requests to cause one or more computer devices (e.g., personal computers, servers or network devices, etc., which may be processors in the computer devices in particular) to perform some or all of the steps of the methods described above for various embodiments of the present invention.
Those skilled in the art will appreciate that all or part of the steps of the various embodiments of the invention recited herein can be implemented by computer programs, which can be stored centrally or in a distributed fashion in one or more computer devices, such as in a readable storage medium. The computer device includes Read-Only Memory (ROM), random-access Memory (Random Access Memory, RAM), programmable Read-Only Memory (Programmable Read-Only Memory, PROM), erasable programmable Read-Only Memory (Erasable Programmable Read Only Memory, EPROM), one-time programmable Read-Only Memory (OTPROM), electrically erasable programmable Read-Only Memory (EEPROM), compact disc Read-Only Memory (CD-ROM) or other optical disc Memory, magnetic disk Memory, magnetic tape Memory, or any other medium capable of being used to carry or store data.
It will be appreciated by persons skilled in the art that the above embodiments are provided for illustration only and not for limitation of the invention, and that variations and modifications of the above described embodiments are intended to fall within the scope of the claims of the invention as long as they fall within the true spirit of the invention.
Claims (9)
1. The utility model provides a multiplexing system of electron outside rear-view mirror and intelligent driving side rear-view camera, includes multiplexing camera to and electron outside rear-view mirror processing module and the domain controller that are connected with multiplexing camera respectively, its characterized in that:
the multiplexing camera is arranged on the side surface of the vehicle body;
the multiplexing camera is in time synchronization with other intelligent driving cameras connected with the domain controller;
The domain controller controls the multiplexing camera and other intelligent driving cameras to be exposed simultaneously, so that all cameras acquire images at the same moment, and the electronic outside rearview mirror processing module controls the exposure time and white balance of the multiplexing camera;
The electronic exterior rearview mirror processing module receives serial image data of the multiplexing camera, divides the serial image data into two identical paths, processes the first path of image data, outputs an exterior rearview mirror image, and transmits the second path of image data to the domain controller;
The domain controller performs frame extraction processing on the second path of image data, so that the image data subjected to frame extraction accords with the condition of a side rear view image recognition algorithm;
Wherein the first path of image data and the second path of image data have frame rates for human eye recognition, and the frame-extracted image data have frame rates for an image recognition algorithm;
The second path of image data is fused with other intelligent driving system cameras, the object monitoring at the rear side is completed through the machine identification of an intelligent driving algorithm, and an instruction is sent to the whole vehicle to complete lane changing early warning and lane changing auxiliary functions.
2. The system for multiplexing an electronic exterior rear view mirror and a front-side rearview camera of claim 1, wherein:
The electronic outside rear view mirror processing module comprises a first deserializer, a first serializer, a first image processor and a controller, wherein the first deserializer is respectively connected with the first serializer and the first image processor, and the first image processor is further connected with the controller;
The first deserializer receives serial image data of the multiplexing camera, deserializes the serial image data and then divides the serial image data into two identical paths, and transmits the two paths of serial image data to the first serializer and the first image processor respectively;
the first image processor processes the first path of image data and then transmits the processed first path of image data to the controller, and the controller processes and outputs the outer rearview mirror image.
3. The system for multiplexing an electronic exterior rear view mirror and a smart side rear view camera of claim 2, wherein:
the domain controller comprises a second deserializer and a second image processor, wherein the second deserializer is respectively connected with the first serializer and the second image processor;
the first serializer transmits the second path of image data to the second deserializer in series, and the second deserializer carries out frame extraction processing on the second path of image data, so that the image data after frame extraction accords with the condition of a side rear view image recognition algorithm.
4. A system for multiplexing an electronic exterior rear view mirror and a front-side rearview camera as recited in claim 3, wherein:
the multiplexing camera comprises an image sensor and a second serializer, wherein the image sensor acquires image data and transmits the image data to the second serializer, and the second serializer transmits the image to the first deserializer in series.
5. The system for multiplexing an electronic exterior rear view mirror and a smart side rear view camera according to any one of claims 1-4, wherein:
the same two-way image data is image data of at least 60 frames in frame rate, and the domain controller decimates the second-way image data into image data of 30 frames in frame rate.
6. A method for multiplexing an electronic exterior rear view mirror and a front-side rearview camera for use in the system of any one of claims 1-5, comprising:
the multiplexing camera is in time synchronization with other intelligent driving cameras connected with the domain controller;
The domain controller is used for controlling the multiplexing camera and other intelligent driving cameras to expose simultaneously, so that all cameras acquire images at the same moment, and further, rear image data of the vehicle body side are obtained;
Controlling the exposure time length and white balance of the multiplexing camera by using an electronic exterior rearview mirror processing module;
dividing the image data into two identical paths;
processing the first path of image data and outputting an outer rearview mirror image;
Performing frame extraction processing on the second path of image data, so that the image data subjected to frame extraction accords with the condition of a side rear view image recognition algorithm;
Wherein the first path of image data and the second path of image data have frame rates for human eye recognition, and the frame-extracted image data have frame rates for an image recognition algorithm;
The second path of image data is fused with other intelligent driving system cameras, the object monitoring at the rear side is completed through the machine identification of an intelligent driving algorithm, and an instruction is sent to the whole vehicle to complete lane changing early warning and lane changing auxiliary functions.
7. The method for multiplexing the electronic outside rear view mirror and the intelligent driving side rear view camera according to claim 6, comprising:
decompressing the first path of image data by using an electronic exterior rearview mirror processing module, and executing at least one of the following image processing operations:
Black level compensation, bad pixel correction, color interpolation, denoising, white balance correction, color correction, gamma correction, color space conversion, color noise removal and edge enhancement;
outputting the processed first path of image data to an outside rearview mirror image;
And serially transmitting the second path of image data to the domain controller.
8. The method for multiplexing the electronic outside rear view mirror and the intelligent driving side rear view camera according to claim 7, comprising:
de-serializing the second path of image data by using a domain controller;
And performing frame extraction processing on the second path of image data, so that the image data subjected to frame extraction accords with the condition of a side rear view image recognition algorithm.
9. The method for multiplexing the electronic outside rear view mirror and the intelligent driving side rear view camera according to any one of claims 6 to 8, wherein:
the same two-way image data is image data of at least 60 frames in frame rate, and the domain controller decimates the second-way image data into image data of 30 frames in frame rate.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210767172.6A CN115297230B (en) | 2022-07-01 | 2022-07-01 | System and method for multiplexing electronic exterior rearview mirror and intelligent driving side rearview camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210767172.6A CN115297230B (en) | 2022-07-01 | 2022-07-01 | System and method for multiplexing electronic exterior rearview mirror and intelligent driving side rearview camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115297230A CN115297230A (en) | 2022-11-04 |
| CN115297230B true CN115297230B (en) | 2024-05-14 |
Family
ID=83822540
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210767172.6A Active CN115297230B (en) | 2022-07-01 | 2022-07-01 | System and method for multiplexing electronic exterior rearview mirror and intelligent driving side rearview camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115297230B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9669761B2 (en) * | 2014-06-13 | 2017-06-06 | Hyundai Mobis Co., Ltd. | Around view monitoring apparatus and method thereof |
| CN207140946U (en) * | 2017-06-16 | 2018-03-27 | 上海赫千电子科技有限公司 | A kind of vehicle-mounted auxiliary drives panoramic picture system |
| CN108556767A (en) * | 2018-03-01 | 2018-09-21 | 李洪运 | A kind of expansible intelligent driving auxiliary system |
| CN109866683A (en) * | 2017-11-20 | 2019-06-11 | 夏普株式会社 | Driving assist apparatuses and methods |
| CN110050462A (en) * | 2016-12-22 | 2019-07-23 | 康奈可关精株式会社 | Image display control apparatus |
| WO2020205629A1 (en) * | 2019-03-29 | 2020-10-08 | Intel Corporation | Autonomous vehicle system |
| CN112550151A (en) * | 2020-11-24 | 2021-03-26 | 新石器慧义知行智驰(北京)科技有限公司 | Camera sharing system of automatic driving vehicle |
| CN112738417A (en) * | 2020-12-24 | 2021-04-30 | 浙江赫千电子科技有限公司 | Data acquisition, storage and display method for vehicle-mounted video |
| CN113706737A (en) * | 2021-10-27 | 2021-11-26 | 北京主线科技有限公司 | Road surface inspection system and method based on automatic driving vehicle |
| CN215219761U (en) * | 2021-05-24 | 2021-12-17 | 深圳市小镜科技有限公司 | AR panorama vehicle event data recorder, vehicle monitored control system and car |
| CN114286019A (en) * | 2021-12-24 | 2022-04-05 | 智己汽车科技有限公司 | Video data processing method, device and equipment |
| CN114419603A (en) * | 2022-02-14 | 2022-04-29 | 广西汽车集团有限公司 | Automatic driving vehicle control method and system and automatic driving vehicle |
| CN114598843A (en) * | 2022-02-09 | 2022-06-07 | 上海赫千电子科技有限公司 | Image processing system and method applied to multi-path cameras of large automobile |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200039448A1 (en) * | 2018-08-01 | 2020-02-06 | Magna Electronics Inc. | Vehicular camera system with dual video outputs |
-
2022
- 2022-07-01 CN CN202210767172.6A patent/CN115297230B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9669761B2 (en) * | 2014-06-13 | 2017-06-06 | Hyundai Mobis Co., Ltd. | Around view monitoring apparatus and method thereof |
| CN110050462A (en) * | 2016-12-22 | 2019-07-23 | 康奈可关精株式会社 | Image display control apparatus |
| CN207140946U (en) * | 2017-06-16 | 2018-03-27 | 上海赫千电子科技有限公司 | A kind of vehicle-mounted auxiliary drives panoramic picture system |
| CN109866683A (en) * | 2017-11-20 | 2019-06-11 | 夏普株式会社 | Driving assist apparatuses and methods |
| CN108556767A (en) * | 2018-03-01 | 2018-09-21 | 李洪运 | A kind of expansible intelligent driving auxiliary system |
| WO2020205629A1 (en) * | 2019-03-29 | 2020-10-08 | Intel Corporation | Autonomous vehicle system |
| CN112550151A (en) * | 2020-11-24 | 2021-03-26 | 新石器慧义知行智驰(北京)科技有限公司 | Camera sharing system of automatic driving vehicle |
| CN112738417A (en) * | 2020-12-24 | 2021-04-30 | 浙江赫千电子科技有限公司 | Data acquisition, storage and display method for vehicle-mounted video |
| CN215219761U (en) * | 2021-05-24 | 2021-12-17 | 深圳市小镜科技有限公司 | AR panorama vehicle event data recorder, vehicle monitored control system and car |
| CN113706737A (en) * | 2021-10-27 | 2021-11-26 | 北京主线科技有限公司 | Road surface inspection system and method based on automatic driving vehicle |
| CN114286019A (en) * | 2021-12-24 | 2022-04-05 | 智己汽车科技有限公司 | Video data processing method, device and equipment |
| CN114598843A (en) * | 2022-02-09 | 2022-06-07 | 上海赫千电子科技有限公司 | Image processing system and method applied to multi-path cameras of large automobile |
| CN114419603A (en) * | 2022-02-14 | 2022-04-29 | 广西汽车集团有限公司 | Automatic driving vehicle control method and system and automatic driving vehicle |
Non-Patent Citations (1)
| Title |
|---|
| 一种自动驾驶汽车系统架构开发与测试验证;龙翔 等;重庆理工大学学报(自然科学);20191215(第12期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115297230A (en) | 2022-11-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20190028651A1 (en) | Imaging device, imaging system, and imaging method | |
| CN109690628B (en) | Image processing apparatus | |
| CN114338958B (en) | Image processing method and related equipment | |
| CN112188093B (en) | Bimodal signal fusion system and method | |
| JPH11161773A (en) | Method for processing picture and device for inputting picture | |
| US20140184737A1 (en) | Driving assistant system and method | |
| CN113781302B (en) | Multi-path image splicing method and system, readable storage medium and unmanned vehicle | |
| CN115297230B (en) | System and method for multiplexing electronic exterior rearview mirror and intelligent driving side rearview camera | |
| CN112188092B (en) | Bimodal signal processing system and method | |
| KR20120055824A (en) | Image acquiring method of vehicle camera system | |
| US20180069998A1 (en) | Imaging apparatus, imaging system, and vehicle | |
| US20040201693A1 (en) | Image processing system | |
| CN115027363A (en) | Automobile intelligent cabin display system and method | |
| US20210097660A1 (en) | Multi-eye camera system, multi-eye photographing camera head, image processing device, multi-eye photographing program and multi-eye photographing method | |
| KR101809727B1 (en) | Surround View Monitoring System and Image Signal Processing Method thereof | |
| US7868913B2 (en) | Apparatus for converting images of vehicle surroundings | |
| JPH118803A (en) | Camera having solid-state image pickup device for image receiving | |
| KR20180028354A (en) | Method for displaying image in multiple view modes | |
| CN114222032A (en) | Vision auxiliary method applied to large vehicle | |
| KR102718457B1 (en) | Apparatus for processing image | |
| CN219428021U (en) | Streaming media rearview mirror image processing circuit and IVI system | |
| CN111492650B (en) | Image preprocessing method and device, image sensor interface, image processing method and device | |
| US20240007595A1 (en) | Camera monitoring system, control method for camera monitoring system, and storage medium | |
| JP5478761B2 (en) | Imaging device | |
| KR102339358B1 (en) | Vehicular device for improving visibility in adverse weather conditions |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |