CN113674422A - Data synchronous acquisition method, control module, system and storage medium - Google Patents
Data synchronous acquisition method, control module, system and storage medium Download PDFInfo
- Publication number
- CN113674422A CN113674422A CN202110992363.8A CN202110992363A CN113674422A CN 113674422 A CN113674422 A CN 113674422A CN 202110992363 A CN202110992363 A CN 202110992363A CN 113674422 A CN113674422 A CN 113674422A
- Authority
- CN
- China
- Prior art keywords
- camera
- shooting
- laser radar
- roller shutter
- rotatable roller
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000001360 synchronised effect Effects 0.000 title claims abstract description 48
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000005096 rolling process Methods 0.000 claims abstract description 16
- 238000004364 calculation method Methods 0.000 claims description 18
- 230000004927 fusion Effects 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 9
- 230000007246 mechanism Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 230000009467 reduction Effects 0.000 description 7
- 238000009434 installation Methods 0.000 description 5
- 239000007787 solid Substances 0.000 description 5
- 230000006872 improvement Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T7/85—Stereo camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Abstract
The invention discloses a data synchronous acquisition method, a control module, a system and a storage medium, wherein the method comprises the following steps: acquiring the rotation direction of the laser radar; generating an adjusting signal for controlling the rotatable roller shutter type camera to rotate to a preset state according to the rotation direction of the laser radar; the stretching direction of the output result of the rotatable roller shutter type camera to the moving object in the preset state is consistent with the stretching direction of the output result of the laser radar to the moving object; acquiring the rotation speed of the laser radar and the shooting parameters of the rotatable roller shutter type camera; setting a shooting frame rate and shooting time of the rotatable roller shutter type camera according to the rotation speed and the shooting parameters; and generating a shooting signal for controlling the rotatable rolling shutter type camera to shoot according to the shooting frame rate and the shooting time. The invention realizes the function of synchronizing the point cloud collection of the pixel points and the laser radar measured by the camera, can greatly reduce the difficulty and the calculated amount of information fusion of the camera and the laser radar, and improves the reliability of the result.
Description
Technical Field
The invention relates to the technical field of automatic driving, in particular to a data synchronous acquisition method, a control module, a system and a storage medium.
Background
At present, a camera and a laser radar are mostly used for sensing targets of automatic driving vehicles, a certain time difference exists between pixel points of the same frame of image of the camera and each point cloud of the laser radar when the point cloud is collected, so that the object outline can deform when the target to be detected moves at a high speed, and the time of each point of two sensors cannot be effectively synchronized.
In practical applications, the roller shutter type exposure camera is applied more due to various factors such as cost and sensitivity, fig. 1 is a schematic diagram of a roller shutter type exposure time in the prior art, specifically, as shown in fig. 1, the roller shutter type exposure camera adopts an exposure mode with independent lines, wherein tVFor a single frame image time, texpAs exposure time, tHFor outputting time for a single line, while tHAnd is also the line exposure misalignment time.
Fig. 2 is a schematic drawing of an image of a horizontally moving object by a camera and a laser radar in the prior art, and fig. 2(a) shows the horizontally moving object, which is square in shape, and solid arrow lines are arranged above the horizontally moving object to indicate the moving direction of the object.
Fig. 2(b) is a schematic diagram of an output result of the camera in the high-speed horizontal motion of the square object shown in fig. 2(a), wherein a solid arrow line is arranged above the camera to indicate a single receiving direction of the camera, a dashed arrow line is arranged to indicate a scanning sequence between adjacent solid arrow lines, and the camera scans for multiple times according to the scanning sequence shown by the arrow lines to obtain the image shown in fig. 2 (b). When a camera which is normally installed shoots a horizontally moving object at a high speed, the object in the output result of the camera is in a parallelogram shape, the shooting result is compared with that in fig. 2(a) to generate oblique stretching, and the stretching amplitude is related to the relative speed and distance of the object. The camera itself cannot measure the relative speed and distance of the object and must rely on other sensors.
Fig. 2(c) is a schematic diagram of the output result of the lidar when the square object shown in fig. 2(a) moves horizontally at a high speed, wherein the solid arrow lines on the left side are used for indicating the single receiving direction of the lidar, the dashed arrow lines are used for indicating the scanning sequence between adjacent solid arrow lines, and the lidar scans for multiple times according to the scanning sequence shown by the arrow lines to obtain the image shown in fig. 2 (c). As shown in fig. 2(a) and 2(c), when the lidar scans around 360 ° generally, resulting in the lidar obtaining a point cloud of a high-speed horizontal moving object, the shape of the object is rectangular in the output result of the lidar, and the output result generates transverse stretching compared with fig. 2(a), and the stretching amplitude is related to the relative speed and distance of the object.
The camera and the horizontal moving object image shot by the laser radar have inconsistent object deformation, information fusion can be carried out only after the object is subjected to shape reduction, accurate object speed is required during reduction, distance information is used as guidance, the calculated amount is large, and the result is uncontrollable.
Therefore, it is necessary to provide a technical solution to reduce the amount of deformation reduction calculation and the calculation difficulty of the camera and the lidar and to achieve the function of synchronizing the point cloud collection of the pixel point and the lidar measured by the camera.
Disclosure of Invention
The present invention is directed to solving at least one of the problems of the prior art. Therefore, the first aspect of the present invention provides a data synchronous acquisition method, including:
acquiring the rotation direction of the laser radar;
generating an adjusting signal for controlling the rotatable roller shutter camera to rotate to a preset state according to the rotating direction; the stretching direction of the output result of the rotatable roller shutter type camera to the moving object in the preset state is consistent with the stretching direction of the output result of the laser radar to the moving object;
acquiring the rotation speed of the laser radar and the shooting parameters of the rotatable roller shutter type camera;
setting the shooting frame rate and the shooting time of the rotatable roller shutter type camera according to the rotation speed and the shooting parameters;
generating a shooting signal for controlling the rotatable roller shutter type camera to shoot according to the shooting frame rate and the shooting time; the rotatable roller shutter type camera can respond to the shooting signal to shoot according to the shooting frame rate and the shooting time, and obtains a target shooting result; and the target shooting result and the scanning result of the laser radar have consistent object deformation.
Further, the generating an adjustment signal for controlling the rotatable roller shutter camera to rotate to a preset state according to the rotation direction includes:
when the laser radar rotates clockwise, generating a first adjusting signal for controlling the rotatable roller shutter type camera to rotate to a first preset state; wherein the first preset state can be obtained by rotating the rotatable roller shutter camera in an initial state by 90 degrees to the left; the output result of the rotatable roller shutter type camera in the initial state is consistent with the display direction of the output result of the laser radar;
when the laser radar rotates anticlockwise, generating a second adjusting signal for controlling the rotatable roller shutter type camera to rotate to a second preset state; the second preset state may be obtained by rotating the rotatable roller shutter camera in the initial state by 90 ° to the right.
Further, the shooting parameters include an effective field angle, all lines, and an effective line; the setting of the photographing frame rate and the photographing time of the rotatable roller shutter camera according to the rotation speed and the photographing parameters includes:
calculating the time of the laser radar scanning the effective field angle of the camera according to the effective field angle and the rotating speed;
calculating a camera frame rate according to the time of all the lines, the effective line and the effective field angle of the camera swept by the laser radar;
obtaining the exposure time of the next frame output by an automatic exposure algorithm based on the camera frame rate;
and calculating the shooting time which can enable the laser radar scanning line to coincide with the middle point of the exposure time of the first line of the camera based on the exposure time of all the lines and the next frame.
Further, based on the exposure time of all the lines and the next frame, calculating the shooting time which can enable the laser radar scanning line to coincide with the middle point of the exposure time of the first line of the camera, and calculating the shooting time by adopting the following formula:
tshot＝-((Vtotal-VFP)tH-texp/2)
wherein, tshotIs the shooting time;
Vtotalall rows of the camera;
VFP is field synchronization front shoulder;
tHoutputting the time for a single row;
texpthe next frame exposure time.
The second aspect of the present invention provides a data synchronous acquisition control module, including:
the first acquisition module is used for acquiring the rotation direction of the laser radar;
the camera adjusting module is used for generating an adjusting signal for controlling the rotatable roller shutter type camera to rotate to a preset state according to the rotating direction; the stretching direction of the output result of the rotatable roller shutter type camera to the moving object in the preset state is consistent with the stretching direction of the output result of the laser radar to the moving object;
the second acquisition module is used for acquiring the rotation speed of the laser radar and the shooting parameters of the rotatable roller shutter type camera;
the shooting parameter setting module is used for setting the shooting frame rate and the shooting time of the rotatable roller shutter type camera according to the rotating speed and the shooting parameters;
the shooting control module is used for generating a shooting signal for controlling the rotatable roller shutter type camera to shoot according to the shooting frame rate and the shooting time; the rotatable roller shutter type camera can respond to the shooting signal to shoot according to the shooting frame rate and the shooting time, and obtains a target shooting result; and the target shooting result and the scanning result of the laser radar have consistent object deformation.
Further, the camera adjustment module includes:
the first adjusting module is used for generating a first adjusting signal for controlling the rotatable roller shutter type camera to rotate to a first preset state when the laser radar rotates clockwise; wherein the first preset state can be obtained by rotating the rotatable roller shutter camera in an initial state by 90 degrees to the left; the output result of the rotatable roller shutter type camera in the initial state is consistent with the display direction of the output result of the laser radar;
the second adjusting module is used for generating a second adjusting signal for controlling the rotatable roller shutter type camera to rotate to a second preset state when the laser radar rotates anticlockwise; the second preset state may be obtained by rotating the rotatable roller shutter camera in the initial state by 90 ° to the right.
Further, the shooting parameters include an effective field angle, all lines, and an effective line; the shooting parameter setting module includes:
the effective field angle calculation module is used for calculating the time of the laser radar scanning the effective field angle of the camera according to the effective field angle and the rotation speed;
the camera frame rate calculation module is used for calculating the camera frame rate according to the total rows, the effective rows and the time of the laser radar scanning the effective field angle of the camera;
the next frame exposure time acquisition module is used for acquiring the next frame exposure time output by the automatic exposure algorithm based on the camera frame rate;
and the shooting time calculation module is used for calculating the shooting time which can enable the scanning line of the laser radar to coincide with the middle point of the exposure time of the first line of the camera based on the exposure time of all the lines and the next frame.
Further, the photographing time calculating module calculates the photographing time using the following formula:
tshot＝-((Vtotal-VFP)tH-texp/2)
wherein, tshotIs the shooting time;
Vtotalall rows of the camera;
VFP is field synchronization front shoulder;
tHoutputting the time for a single row;
texpis the next oneFrame exposure time.
The third aspect of the invention provides a data synchronous acquisition system, which comprises a laser radar, a rotatable and rotatable roller shutter type camera and a data synchronous acquisition control module of the second aspect of the invention;
the quantity of rotatable roll of curtain formula camera is one or more, and is a plurality of rotatable roll of curtain formula camera is followed laser radar's circumference distributes, laser radar with data synchronization acquisition control module connects, rotatable roll of curtain formula camera with data synchronization acquisition control module connects.
A fourth aspect of the present invention provides an electronic device, where the electronic device includes a processor and a memory, where the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the data synchronous acquisition method provided in the first aspect of the present invention.
A fifth aspect of the present invention provides a computer-readable storage medium, where at least one instruction, at least one program, a code set, or a set of instructions is stored in the storage medium, and the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by a processor to implement the data synchronous acquisition method provided in the first aspect of the present invention.
The data synchronous acquisition method, the control module, the system and the storage medium provided by the invention apply a camera installation method rotating by 90 degrees, so that the stretching direction of a camera shooting result relative to a moving object is consistent with the stretching direction of a laser radar scanning result relative to the moving object; by setting camera parameters, the picture width of the camera and the laser radar are completely the same in the same field angle, and the pixels of the camera and the laser radar can be matched with the point cloud; by setting shooting time, the rolling shutter type camera and the laser radar can realize high-precision time synchronization close to a pixel level; based on the comprehensive application of the three aspects of improvement, the motion characteristics, the object positions and the time information of the camera and the laser radar are unified, the function that the camera measures the point cloud collection synchronization of the pixel points and the laser radar is realized, the difficulty and the calculated amount of information fusion of the camera and the laser radar can be greatly reduced, and the result reliability is improved.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic diagram of a rolling shutter exposure time in the prior art;
FIG. 2 is a schematic drawing of a camera and lidar for stretching an image of a horizontally moving object in the prior art;
fig. 3 is a flowchart of a data synchronous acquisition method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a camera rotation provided by an embodiment of the invention;
FIG. 5 is a flowchart of step S104 provided by the embodiment of the present invention;
fig. 6 is a block diagram of a structure of a data synchronous acquisition control module according to an embodiment of the present invention;
fig. 7 is an installation schematic diagram of a data synchronous acquisition system according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. Examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout.
Examples
In order to reduce the deformation reduction calculated amount and the reduction difficulty of a camera and a laser radar and realize the function of synchronous point cloud acquisition of a pixel point and the laser radar measured by the camera, the embodiment of the invention provides a data synchronous acquisition method which can be used for reducing the subsequent information fusion calculated amount and improving the result reliability.
Fig. 3 is a flowchart of a data synchronous acquisition method provided by an embodiment of the present invention, and the present specification provides the method operation steps as described in the embodiment or the flowchart, but more or less operation steps may be included based on conventional or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. In practice, the system or server product may be implemented in a sequential or parallel manner (e.g., parallel processor or multi-threaded environment) according to the embodiments or methods shown in the figures. As shown in fig. 3, the method may include the following steps:
s101: acquiring the rotation direction of the laser radar;
specifically, the rotation direction is an inherent property of the mechanical lidar, and the rotation direction of the lidar may be clockwise rotation or counterclockwise rotation.
S102: generating an adjusting signal for controlling the rotatable roller shutter type camera to rotate to a preset state according to the rotation direction of the laser radar;
and the stretching direction of the output result of the rotatable roller shutter type camera to the moving object in the preset state is consistent with the stretching direction of the output result of the laser radar to the moving object.
It should be noted that the rotation direction of the lidar is only used to determine an adjustment manner (including the rotation direction and the rotation angle) for rotating the rolling-up camera from the initial state to the preset state, and the camera does not rotate during the operation process and is not affected by the rotation direction of the lidar.
In some embodiments, generating an adjustment signal for controlling the rotatable roller shutter camera to rotate to a preset state according to a rotation direction of the lidar includes:
when the laser radar rotates clockwise, generating a first adjusting signal for controlling the rotatable roller shutter type camera to rotate to a first preset state; the first preset state can be obtained by rotating the rotatable roller shutter type camera in the initial state by 90 degrees to the left; the output result of the rotatable roller shutter type camera is consistent with the display direction of the output result of the laser radar in the initial state; fig. 4 is a schematic view of camera rotation provided by an embodiment of the present invention, in which fig. 4(a) shows a rotatable rolling shutter camera in an initial state, fig. 4(b) shows the rotatable rolling shutter camera after rotation, and an arc with an arrow between fig. 4(a) and fig. 4(b) is used to indicate a rotation direction of the rotatable rolling shutter camera.
When the laser radar rotates anticlockwise, generating a second adjusting signal for controlling the rotatable roller shutter type camera to rotate to a second preset state; the second preset state can be obtained by rotating the rotatable roller shutter type camera at the initial state by 90 degrees rightwards;
when the laser radar rotates anticlockwise, generating a second rotation signal; the second rotation signal is used for controlling the rotatable roller shutter camera to rotate 90 degrees rightwards from the initial state;
and the display direction of the camera picture output by the rotatable roller shutter type camera in the initial state is consistent with the display direction of the scanning result of the laser radar.
S103: acquiring the rotation speed m of the laser radar and the shooting parameters of the rotatable roller shutter type camera; wherein the shooting parameters comprise an effective field angle p and all rows VtotalAnd an active row;
wherein all rows VtotalIs the corresponding line between two field sync pulses, and the active line V is the corresponding line of the display timing segment.
Specifically, the unit of the effective field angle p is "° (degree)", the size of the effective field angle p is usually 90 ° and 120 °, and the value of the effective field angle p may be set to other values, for example, 70 ° and 110 ° according to actual needs, which is not limited in this embodiment.
Specifically, the unit of the lidar rotation speed is "r/s (circle per second)", the size of the lidar rotation speed m is usually 10r/s, and the effective field angle p may be other values, for example, 70 ° and 110 ° according to actual needs, which is not limited in this embodiment.
S104: setting a shooting frame rate Fs and a shooting time t of the rotatable roller shutter camera according to the rotation speed m and the shooting parametersshot；
By modifying the parameter setting of the camera, the picture width of the camera and the laser radar can be completely the same in the same field angle, and the pixels of the camera and the laser radar can be matched with the point cloud. Fig. 5 is a flowchart of step S104 according to an embodiment of the present invention, and specifically as shown in fig. 5, in some embodiments, step S104 includes the following steps:
s1041: calculating the time t of the laser radar scanning the effective field angle of the camera according to the effective field angle p and the rotation speed msweep；
Specifically, the calculation formula of the time for the laser radar to sweep through the effective field angle of the camera is tsweep＝p/(360m)
S1042: according to all rows VtotalThe effective line V and the time t for the laser radar to sweep through the effective field angle of the camerasweepCalculating a camera frame rate Fs;
specifically, the calculation formula of the camera frame rate is Fs ═ V/(V)total/tsweep)。
S1043: obtaining the next frame exposure time t output by the automatic exposure algorithm based on the frame rate Fs of the cameraexp；
S1044: based on all rows VtotalNext frame exposure time texpCalculating the shooting time t which can make the scanning line of the laser radar coincide with the middle point of the exposure time of the first line of the camerashot。
In some embodiments, the shooting time t is calculated using the following formulashot：
tshot＝-((Vtotal-VFP)tH-texp/2)
Wherein, tshotIs the shooting time;
Vtotalall rows of the camera;
VFP is field synchronization front shoulder;
tHoutputting the time for a single row;
texpthe next frame exposure time.
The minus sign in the photographing time calculation formula means that the calculated photographing time is a time that needs to be advanced.
The set shooting time is calculated according to the method, and the rolling shutter type camera and the laser radar can realize high-precision time synchronization close to a pixel level.
The data synchronous acquisition method provided by the embodiment of the invention unifies the motion characteristics, object positions and time information of the rotatable camera and the laser radar, can greatly reduce the deformation reduction calculation amount and reduction difficulty of the camera and the laser radar, realizes the function of synchronous point cloud acquisition of the pixel points and the laser radar measured by the camera, is beneficial to reducing the subsequent information fusion calculation amount, and improves the result reliability.
S105: generating a photographing signal for controlling photographing of the rotatable roller shutter camera according to the photographing frame rate and the photographing time;
the rotatable roller shutter type camera can respond to a shooting signal to shoot according to a shooting frame rate and shooting time, and obtains a target shooting result; the target shooting result and the scanning result of the laser radar have consistent object deformation.
Fig. 6 is a block diagram of a structure of a data synchronous acquisition control module according to an embodiment of the present invention, and specifically, as shown in fig. 6, the data synchronous acquisition control module according to the embodiment of the present invention includes the following modules:
a first obtaining module 201, configured to obtain a rotation direction of the laser radar;
the camera adjusting module 202 is configured to generate an adjusting signal for controlling the rotatable roller shutter camera to rotate to a preset state according to the rotation direction of the laser radar;
and the stretching direction of the output result of the rotatable roller shutter type camera to the moving object in the preset state is consistent with the stretching direction of the output result of the laser radar to the moving object.
A second obtaining module 203, configured to obtain a rotation speed of the laser radar and shooting parameters of the rotatable roller shutter camera;
a shooting parameter setting module 204, configured to set a shooting frame rate and a shooting time of the rotatable roller shutter camera according to the rotation speed and the shooting parameters;
a photographing control module 205 for generating a photographing signal for controlling photographing of the rotatable roller shutter camera according to a photographing frame rate and a photographing time;
the rotatable roller shutter type camera can respond to a shooting signal to shoot according to a shooting frame rate and shooting time, and obtains a target shooting result; the target shooting result and the scanning result of the laser radar have consistent object deformation.
In some embodiments, the camera adjustment module 202 includes the following modules:
the first adjusting module is used for generating a first adjusting signal for controlling the rotatable roller shutter type camera to rotate to a first preset state when the laser radar rotates clockwise; the first preset state can be obtained by rotating the rotatable roller shutter type camera in the initial state by 90 degrees to the left; the output result of the rotatable roller shutter type camera is consistent with the display direction of the output result of the laser radar in the initial state;
the second adjusting module is used for generating a second adjusting signal for controlling the rotatable roller shutter type camera to rotate to a second preset state when the laser radar rotates anticlockwise; the second preset state can be obtained by rotating the rotatable roller shutter camera in the initial state by 90 ° to the right.
In some embodiments, the shooting parameters include effective field angle, all rows VtotalAnd an active row V; the shooting parameter setting module 204 includes:
an effective field angle calculation module for calculating the time t for the laser radar to sweep the effective field angle of the camera according to the effective field angle p and the rotation speed msweep＝p/(360m)；
A camera frame rate calculation module for calculating a frame rate according to all rows VtotalThe effective line V and the time t for the laser radar to sweep through the effective field angle of the camerasweepCalculating the frame rate Fs of the camera as V/(V)total/tsweep)；
A next frame exposure time acquisition module for obtaining the next frame exposure time t output by the automatic exposure algorithm based on the camera frame rate Fsexp；
A photographing time calculation module for calculating a photographing time based on all the lines VtotalNext frame exposure time texpCalculating the shooting time t which can make the scanning line of the laser radar coincide with the middle point of the exposure time of the first line of the camerashot. The shooting time calculation module calculates the shooting time t by adopting the following formulashot：
tshot＝-((Vtotal-VFP)tH-texp/2)
Wherein, tshotIs the shooting time;
Vtotalall rows of the camera;
VFP is field synchronization front shoulder;
tHoutputting the time for a single row;
texpthe next frame exposure time.
Fig. 7 is an installation schematic diagram of a data synchronous acquisition system according to an embodiment of the present invention, specifically, as shown in fig. 7, the data synchronous acquisition system includes a laser radar, a rotatable and rotatable rolling shutter camera, and a data synchronous acquisition control module disclosed in the foregoing embodiment;
in some embodiments, the number of the rotatable rolling shutter type cameras is one, the rotatable rolling shutter type cameras are arranged along the circumferential direction of the laser radar, the laser radar is synchronously connected with the data synchronous acquisition device through a PPS synchronous signal, and the rotatable rolling shutter type cameras are connected with the data synchronous acquisition control module.
In some embodiments, the number of the rotatable rolling shutter type cameras is multiple, the plurality of rotatable rolling shutter type cameras are distributed along the circumferential direction of the laser radar, and the laser radar and the data synchronous acquisition deviceThe rotatable roller shutter type camera is connected with the data synchronous acquisition control module through PPS synchronous signal synchronous connection. The data synchronous acquisition system can be suitable for the combined reception of various cameras with different resolutions and different field angles and a laser radar. For example, fig. 7 shows a synchronous data acquisition system in which there are 3 rotatable rolling shutter cameras with different effective field angles, camera 1, camera 2, and camera 3, and the effective field angle of camera 1 is p1Angle of view of camera 2 is p2Angle of view of camera 3 is p3°，p1>p2>p3。
Specifically, the data synchronous acquisition control module may be an FPGA chip or an MCU chip.
Specifically, the rotatable roller shutter camera includes a rotating mechanism and a roller shutter camera, and the roller shutter camera is rotatably connected to the rotating mechanism.
Theoretically, it is also feasible to adopt an adjustment mode in which the rotating mechanism is manually acted on by a person to rotate the roller shutter camera to the preset state, and certainly, it is also feasible to adjust the camera from the initial state to the preset state without the rotating mechanism. However, the above scheme is not convenient and fast enough in operation, the installation precision is difficult to guarantee, and in consideration of convenience and accuracy, an automatic control adjustment mode is preferably adopted in practical application, namely, an adjustment mode that the rotating mechanism is controlled to automatically rotate to a preset state through the data synchronous acquisition control module is adopted.
When an automatic control adjusting mode is adopted, the rotating mechanism can be electrically connected with the data synchronous acquisition control module, and the rotating mechanism has the function of driving the roller shutter camera to rotate by a preset angle in a preset direction according to the action of an adjusting signal.
It should be noted that the rotating mechanism may have various implementations, and the present embodiment is not limited to this.
An embodiment of the present invention further provides an electronic device, where the electronic device includes a processor and a memory, where the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the data synchronous acquisition method in the method embodiment.
The embodiment of the present invention further provides a storage medium, where the storage medium may be disposed in a server to store at least one instruction, at least one program, a code set, or a set of instructions related to implementing the data synchronous acquisition method in the method embodiment, and the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by the processor to implement the data synchronous acquisition method provided in the method embodiment.
Alternatively, in this embodiment, the storage medium may be located in at least one network server of a plurality of network servers of a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
It can be seen from the above embodiments of the method, apparatus, system and storage medium for synchronous data acquisition provided by the present invention that a camera installation method rotating by 90 ° is applied in the embodiments of the present invention, so that the stretching direction of the camera shooting result relative to the moving object is consistent with the stretching direction of the laser radar scanning result relative to the moving object; by setting camera parameters, the picture width of the camera and the laser radar are completely the same in the same field angle, and the pixels of the camera and the laser radar can be matched with the point cloud; by setting shooting time, the rolling shutter type camera and the laser radar can realize high-precision time synchronization close to a pixel level; based on the comprehensive application of the three aspects of improvement, the motion characteristics, the object positions and the time information of the camera and the laser radar are unified, the function that the camera measures the point cloud collection synchronization of the pixel points and the laser radar is realized, the difficulty and the calculated amount of information fusion of the camera and the laser radar can be greatly reduced, and the result reliability is improved.
It should be noted that: the precedence order of the above embodiments of the present invention is only for description, and does not represent the merits of the embodiments. And specific embodiments thereof have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the device and server embodiments, since they are substantially similar to the method embodiments, the description is simple, and the relevant points can be referred to the partial description of the method embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A data synchronous acquisition method is characterized by comprising the following steps:
acquiring the rotation direction of the laser radar;
generating an adjusting signal for controlling the rotatable roller shutter camera to rotate to a preset state according to the rotating direction; the stretching direction of the output result of the rotatable roller shutter type camera to the moving object in the preset state is consistent with the stretching direction of the output result of the laser radar to the moving object;
acquiring the rotation speed of the laser radar and the shooting parameters of the rotatable roller shutter type camera;
setting the shooting frame rate and the shooting time of the rotatable roller shutter type camera according to the rotation speed and the shooting parameters;
generating a shooting signal for controlling the rotatable roller shutter type camera to shoot according to the shooting frame rate and the shooting time; the rotatable roller shutter type camera can respond to the shooting signal to shoot according to the shooting frame rate and the shooting time, and obtains a target shooting result; and the target shooting result and the scanning result of the laser radar have consistent object deformation.
2. The method for synchronously acquiring data according to claim 1, wherein the generating an adjustment signal for controlling the rotatable rolling-up camera to rotate to a preset state according to the rotation direction comprises:
when the laser radar rotates clockwise, generating a first adjusting signal for controlling the rotatable roller shutter type camera to rotate to a first preset state; wherein the first preset state can be obtained by rotating the rotatable roller shutter camera in an initial state by 90 degrees to the left; the output result of the rotatable roller shutter type camera in the initial state is consistent with the display direction of the output result of the laser radar;
when the laser radar rotates anticlockwise, generating a second adjusting signal for controlling the rotatable roller shutter type camera to rotate to a second preset state; the second preset state may be obtained by rotating the rotatable roller shutter camera in the initial state by 90 ° to the right.
3. The data synchronous acquisition method according to claim 1, wherein the shooting parameters include an effective field angle, all lines, and effective lines; the setting of the photographing frame rate and the photographing time of the rotatable roller shutter camera according to the rotation speed and the photographing parameters includes:
calculating the time of the laser radar scanning the effective field angle of the camera according to the effective field angle and the rotating speed;
calculating a camera frame rate according to the time of all the lines, the effective line and the effective field angle of the camera swept by the laser radar;
obtaining the exposure time of the next frame output by an automatic exposure algorithm based on the camera frame rate;
and calculating the shooting time which can enable the laser radar scanning line to coincide with the middle point of the exposure time of the first line of the camera based on the exposure time of all the lines and the next frame.
4. The method according to claim 3, wherein the step of calculating the shooting time for the laser radar scanning line to coincide with the middle point of the exposure time of the first line of the camera based on the exposure time of all lines and the next frame is performed by using the following formula:
tshot＝-((Vtotal-VFP)tH-texp/2)
wherein, tshotIs the shooting time;
Vtotalall rows of the camera;
VFP is field synchronization front shoulder;
tHoutputting the time for a single row;
texpthe next frame exposure time.
5. A data synchronous acquisition control module, comprising:
the first acquisition module is used for acquiring the rotation direction of the laser radar;
the camera adjusting module is used for generating an adjusting signal for controlling the rotatable roller shutter type camera to rotate to a preset state according to the rotating direction; the stretching direction of the output result of the rotatable roller shutter type camera to the moving object in the preset state is consistent with the stretching direction of the output result of the laser radar to the moving object;
the second acquisition module is used for acquiring the rotation speed of the laser radar and the shooting parameters of the rotatable roller shutter type camera;
the shooting parameter setting module is used for setting the shooting frame rate and the shooting time of the rotatable roller shutter type camera according to the rotating speed and the shooting parameters;
the shooting control module is used for generating a shooting signal for controlling the rotatable roller shutter type camera to shoot according to the shooting frame rate and the shooting time; the rotatable roller shutter type camera can respond to the shooting signal to shoot according to the shooting frame rate and the shooting time, and obtains a target shooting result; and the target shooting result and the scanning result of the laser radar have consistent object deformation.
6. The synchronous data acquisition control module of claim 5, wherein the camera adjustment module comprises:
the first adjusting module is used for generating a first adjusting signal for controlling the rotatable roller shutter type camera to rotate to a first preset state when the laser radar rotates clockwise; wherein the first preset state can be obtained by rotating the rotatable roller shutter camera in an initial state by 90 degrees to the left; the output result of the rotatable roller shutter type camera in the initial state is consistent with the display direction of the output result of the laser radar;
the second adjusting module is used for generating a second adjusting signal for controlling the rotatable roller shutter type camera to rotate to a second preset state when the laser radar rotates anticlockwise; the second preset state may be obtained by rotating the rotatable roller shutter camera in the initial state by 90 ° to the right.
7. The module of claim 5, wherein the capture parameters include effective field angle, all rows, and effective rows; the shooting parameter setting module includes:
the effective field angle calculation module is used for calculating the time of the laser radar scanning the effective field angle of the camera according to the effective field angle and the rotation speed;
the camera frame rate calculation module is used for calculating the camera frame rate according to the total rows, the effective rows and the time of the laser radar scanning the effective field angle of the camera;
the next frame exposure time acquisition module is used for acquiring the next frame exposure time output by the automatic exposure algorithm based on the camera frame rate;
and the shooting time calculation module is used for calculating the shooting time which can enable the scanning line of the laser radar to coincide with the middle point of the exposure time of the first line of the camera based on the exposure time of all the lines and the next frame.
8. The module of claim 7, wherein the shooting time calculation module calculates the shooting time using the following formula:
tshot＝-((Vtotal-VFP)tH-texp/2)
wherein, tshotIs the shooting time;
Vtotalall rows of the camera;
VFP is field synchronization front shoulder;
tHoutputting the time for a single row;
texpthe next frame exposure time.
9. A data synchronous acquisition system, which is characterized by comprising a laser radar, a rotatable and rotatable rolling shutter type camera and a data synchronous acquisition control module of any one of claims 5 to 8;
the quantity of rotatable roll of curtain formula camera is one or more, and is a plurality of rotatable roll of curtain formula camera is followed lidar's circumference distributes, lidar with data synchronization collection system connects, rotatable roll of curtain formula camera with data synchronization collection system connects.
10. A computer readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by a processor to implement the method of synchronous data acquisition according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110992363.8A CN113674422A (en) | 2021-08-27 | 2021-08-27 | Data synchronous acquisition method, control module, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110992363.8A CN113674422A (en) | 2021-08-27 | 2021-08-27 | Data synchronous acquisition method, control module, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113674422A true CN113674422A (en) | 2021-11-19 |
Family
ID=78546739
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110992363.8A Pending CN113674422A (en) | 2021-08-27 | 2021-08-27 | Data synchronous acquisition method, control module, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113674422A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114338951A (en) * | 2021-12-30 | 2022-04-12 | 智道网联科技(北京)有限公司 | Sensor synchronization method, device and system and vehicle |
| CN114339157A (en) * | 2021-12-30 | 2022-04-12 | 福州大学 | Multi-camera real-time splicing system and method with adjustable observation area |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018091970A1 (en) * | 2016-11-16 | 2018-05-24 | Innoviz Technologies Ltd. | Lidar systems and methods |
| KR20200023974A (en) * | 2018-08-27 | 2020-03-06 | 국방과학연구소 | Method and apparatus for synchronization of rotating lidar and multiple cameras |
| WO2020088165A1 (en) * | 2018-10-30 | 2020-05-07 | 上海禾赛光电科技有限公司 | Camera and laser radar fusion system |
| CN111435162A (en) * | 2020-03-03 | 2020-07-21 | 深圳市镭神智能系统有限公司 | Laser radar and camera synchronization method, device, equipment and storage medium |
| CN111694015A (en) * | 2020-05-29 | 2020-09-22 | 广州大学 | Two-dimensional scanning method, system, device and medium based on laser radar |
| WO2020242945A1 (en) * | 2019-05-24 | 2020-12-03 | Mobileye Vision Technologies Ltd. | Systems and methods for vehicle navigation based on image analysis |
| US20210003712A1 (en) * | 2019-07-05 | 2021-01-07 | DeepMap Inc. | Lidar-to-camera transformation during sensor calibration for autonomous vehicles |
| CN112861660A (en) * | 2021-01-22 | 2021-05-28 | 上海西井信息科技有限公司 | Laser radar array and camera synchronization device, method, equipment and storage medium |
| CN113014812A (en) * | 2021-03-01 | 2021-06-22 | 中电海康集团有限公司 | Camera and laser radar time synchronization control system |
| WO2021138616A1 (en) * | 2020-01-03 | 2021-07-08 | Mobileye Vision Technologies Ltd. | Systems and methods for vehicle navigation |
| CN113219479A (en) * | 2021-05-13 | 2021-08-06 | 环宇智行科技(苏州)有限公司 | Camera and laser radar synchronization method and system of intelligent driving control system |
-
2021
- 2021-08-27 CN CN202110992363.8A patent/CN113674422A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018091970A1 (en) * | 2016-11-16 | 2018-05-24 | Innoviz Technologies Ltd. | Lidar systems and methods |
| KR20200023974A (en) * | 2018-08-27 | 2020-03-06 | 국방과학연구소 | Method and apparatus for synchronization of rotating lidar and multiple cameras |
| WO2020088165A1 (en) * | 2018-10-30 | 2020-05-07 | 上海禾赛光电科技有限公司 | Camera and laser radar fusion system |
| WO2020242945A1 (en) * | 2019-05-24 | 2020-12-03 | Mobileye Vision Technologies Ltd. | Systems and methods for vehicle navigation based on image analysis |
| US20210003712A1 (en) * | 2019-07-05 | 2021-01-07 | DeepMap Inc. | Lidar-to-camera transformation during sensor calibration for autonomous vehicles |
| WO2021138616A1 (en) * | 2020-01-03 | 2021-07-08 | Mobileye Vision Technologies Ltd. | Systems and methods for vehicle navigation |
| CN111435162A (en) * | 2020-03-03 | 2020-07-21 | 深圳市镭神智能系统有限公司 | Laser radar and camera synchronization method, device, equipment and storage medium |
| CN111694015A (en) * | 2020-05-29 | 2020-09-22 | 广州大学 | Two-dimensional scanning method, system, device and medium based on laser radar |
| CN112861660A (en) * | 2021-01-22 | 2021-05-28 | 上海西井信息科技有限公司 | Laser radar array and camera synchronization device, method, equipment and storage medium |
| CN113014812A (en) * | 2021-03-01 | 2021-06-22 | 中电海康集团有限公司 | Camera and laser radar time synchronization control system |
| CN113219479A (en) * | 2021-05-13 | 2021-08-06 | 环宇智行科技(苏州)有限公司 | Camera and laser radar synchronization method and system of intelligent driving control system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114338951A (en) * | 2021-12-30 | 2022-04-12 | 智道网联科技(北京)有限公司 | Sensor synchronization method, device and system and vehicle |
| CN114339157A (en) * | 2021-12-30 | 2022-04-12 | 福州大学 | Multi-camera real-time splicing system and method with adjustable observation area |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11375173B2 (en) | Auto-alignment of image sensors in a multi-camera system | |
| JP6717534B2 (en) | Stereo support with rolling shutter | |
| US10055816B2 (en) | Target-less auto-alignment of image sensors in a multi-camera system | |
| US11388339B2 (en) | Anti-shake method for panoramic video, and portable terminal | |
| CN113674422A (en) | Data synchronous acquisition method, control module, system and storage medium | |
| US8441520B2 (en) | Primary and auxiliary image capture devcies for image processing and related methods | |
| EP2253131B1 (en) | Panoramic camera with multiple image sensors using timed shutters | |
| US10728529B2 (en) | Synchronization of frame captures from multiple cameras with different fields of capture | |
| CN110969097A (en) | Linkage tracking control method, equipment and storage device for monitored target | |
| CN102547133A (en) | Imaging apparatus and shutter operation correcting method | |
| WO2019184410A1 (en) | Method and apparatus for measuring distortion parameters of virtual reality device, and measuring system | |
| WO2013096331A1 (en) | Method and system for image centering during zooming | |
| US11835627B2 (en) | Image capturing method and device | |
| CN113242360A (en) | Linear array camera and line frequency calculation method and device thereof | |
| CN110749311B (en) | Positioning method, positioning device and storage medium | |
| CN108260360B (en) | Scene depth calculation method and device and terminal | |
| CN110991550A (en) | Video monitoring method and device, electronic equipment and storage medium | |
| US20220038646A1 (en) | Synchronized camera system having two different cameras | |
| US10893201B2 (en) | Video stabilization method with non-linear frame motion correction in three axes | |
| CN112258405A (en) | Linear array swinging CCD camera image non-control point geometric correction method | |
| US11659270B2 (en) | Image processing device for detecting horizontal direction of photographing angle | |
| CN114612360B (en) | Video fusion method and system based on motion model | |
| JP5545486B2 (en) | Method and apparatus for measuring shape of coating pattern applied to sheet | |
| KR101348855B1 (en) | Method, apparatus and image capturing system for real-time image stabilization with reduced search range using motion sensors | |
| JP5950197B2 (en) | Video signal processing device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |