CN115797442A - Simulation image re-injection method of target position and related equipment - Google Patents

Abstract

The present application relates to a simulation image re-injection method of a target position and related equipment, wherein the method includes: acquiring first camera data of a first camera set of a first vehicle and second camera data of a second camera set of a second vehicle ; Determine at least one second camera adjacent to the target position according to the external parameter matrix of the target camera and the external parameter matrix of each second camera; A camera data forms a simulated image matching the target position; the simulated image is injected back into the data processing unit of the second vehicle. In the present application, by using the first camera data of the first vehicle to form a simulation image at the target position of the second vehicle, the second camera data can be further enriched, the adaptability can be enhanced, and the application range can be expanded.

Description

Simulation image re-injection method of target position and related equipment

technical field

The present application relates to the field of computer technology, in particular to a simulation image reinjection method of a target position and related equipment.

Background technique

At present, unmanned driving technology is booming, which relies most on the data collected in real situations. In the process of algorithm development, the data collected in real situations can be injected back into the controller, which can verify the effect of the algorithm and improve the efficiency of algorithm development and verification.

Specifically, an algorithm (such as a neural network for machine learning) is set in the controller. After the algorithm development is completed, the camera on the car can inject the collected video data into the controller. The algorithm of the controller can process the collected video data to obtain output results, so as to realize various functions such as target recognition.

In the process of algorithm development, the algorithm (such as neural network) needs to be trained and verified. At this time, it is necessary to inject various video data into the algorithm of the controller. Can be simulated. However, in the prior art, when it is necessary to inject video data into the controller of a new model vehicle, the camera of the new model vehicle needs to actually collect real video data, or to form simulated video data specifically for the camera of the new model. When new models of vehicles are injected with video data, there is often not much video data, and the training and verification effects cannot be achieved well.

Contents of the invention

In view of this, this application proposes a simulation image re-injection method of the target position and related equipment, which can use the first camera data of the first vehicle to form a simulation image at the target position of the second vehicle, and further enrich the second vehicle. The second camera data, enhance the adaptability of the second camera data, and expand the application range of the second camera data.

According to one aspect of the present application, a simulation image re-injection method of a target position is provided, the method comprising: acquiring the first camera data of the first camera set of the first vehicle and the first camera data of the second camera set of the second vehicle Two camera data, the first camera data includes an external parameter matrix of each first camera, the first camera set includes a preset target camera, the second camera set includes a plurality of second cameras, and the target camera The position of the target is the same as the preset target position on the second vehicle, the target position is different from the position of each of the second cameras, and the second camera data includes the external parameter matrix of the second camera and the second The real image captured by the camera; according to the extrinsic parameter matrix of the target camera and the extrinsic parameter matrix of each of the second cameras, at least one second camera adjacent to the target position is determined; based on the at least one second camera The real image corresponding to each camera, the depth map, and the first camera data corresponding to the target camera form a simulation image matching the target position; inject the simulation image back into the data processing unit of the second vehicle .

According to still another aspect of the present application, there is provided a simulated image reinjection device of a target camera, the simulated image reinjection device of the target camera includes: a camera data acquisition module, configured to acquire the data of the first camera set of the first vehicle The first camera data and the second camera data of the second camera set of the second vehicle, the first camera data include the extrinsic parameter matrix of each first camera, the first camera set includes a preset target camera, the The second camera set includes a plurality of second cameras, the position of the target camera is the same as the preset target position on the second vehicle, the target position is different from the position of each of the second cameras, and the second The camera data includes the extrinsic parameter matrix of the second camera and the real image captured by the second camera; the camera determination module is configured to determine according to the extrinsic parameter matrix of the target camera and the extrinsic parameter matrices of each of the second cameras At least one second camera adjacent to the target position; an image forming module, configured to form an image corresponding to the first camera data corresponding to the at least one second camera based on the real image, the depth map, and the first camera data corresponding to the target camera. A simulation image matching the target position; a reinjection module, configured to reinject the simulation image into the data processing unit of the second vehicle.

By acquiring the first camera data of the first camera set of the first vehicle and the second camera data of the second camera set of the second vehicle, then according to the extrinsic parameter matrix of the target camera and the extrinsic parameter matrix of each second camera At least one second camera adjacent to the position, and then based on the real image corresponding to each of the at least one second camera, the depth map, and the first camera data corresponding to the target camera, a simulation image matching the target position is formed, and finally the simulated The image is re-injected to the data processing unit of the second vehicle. According to various aspects of the present application, the first camera data of the first vehicle can be used to form a simulation image at the target position of the second vehicle, and the second camera data of the second vehicle can be further enriched. , enhance the adaptability of the second camera data, and expand the application range of the second camera data.

Description of drawings

The technical solutions and other beneficial effects of the present application will be apparent through the detailed description of the specific embodiments of the present application below in conjunction with the accompanying drawings.

FIG. 1 shows a flowchart of a method for reinjection of a simulated image at a target location according to an embodiment of the present application.

Fig. 2 shows a schematic diagram of a target position and a target camera according to an embodiment of the present application.

Fig. 3 shows a block diagram of a device for reinjecting a simulated image of a target camera according to an embodiment of the present application.

Fig. 4 shows a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Apparently, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts belong to the scope of protection of this application.

In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", " Orientation indicated by rear, left, right, vertical, horizontal, top, bottom, inside, outside, clockwise, counterclockwise, etc. The positional relationship is based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the application and simplifying the description, and does not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, Therefore, it should not be construed as limiting the application. In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of said features. In the description of the present application, "plurality" means two or more, unless otherwise specifically defined.

In the description of this application, it should be noted that unless otherwise specified and limited, the terms "installation", "connection", and "connection" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection. Connected, or integrally connected; may be mechanically connected, may also be electrically connected or may communicate with each other; may be directly connected, or indirectly connected through an intermediary, may be the internal communication of two components or the interaction of two components relation. Those of ordinary skill in the art can understand the specific meanings of the above terms in this application according to specific situations.

The following disclosure provides many different implementations or examples for implementing different structures of the present application. To simplify the disclosure of the present application, components and arrangements of specific examples are described below. Of course, they are examples only and are not intended to limit the application. Furthermore, the present application may repeat reference numerals and/or reference letters in various instances, such repetition is for simplicity and clarity and does not in itself indicate a relationship between the various embodiments and/or arrangements discussed. In addition, various specific process and material examples are provided herein, but one of ordinary skill in the art may recognize the use of other processes and/or the use of other materials. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail in order to highlight the gist of the present application.

FIG. 1 shows a flowchart of a method for reinjection of a simulated image at a target location according to an embodiment of the present application. As shown in Figure 1, the simulated image reinjection method of the target position of the present application may include:

Step S1: Obtain the first camera data of the first camera set of the first vehicle and the second camera data of the second camera set of the second vehicle, the first camera data includes the external parameter matrix of each first camera, the The first camera set includes a preset target camera, the second camera set includes a plurality of second cameras, the position of the target camera is the same as the preset target position on the second vehicle, and the target position is the same as each The positions of the second cameras are different, and the second camera data includes an extrinsic parameter matrix of the second camera and a real image captured by the second camera;

Wherein, the plurality of first cameras may be installed on the first vehicle, and there may be one or more than one first vehicle. Correspondingly, the plurality of second cameras may be installed on the second vehicle, and there may be one or more than one second vehicle. It can be understood that the present application does not limit the quantity of the first vehicle and the quantity of the second vehicle.

Wherein, the first camera data may be a data set, and the first camera data may include parameters such as an extrinsic parameter matrix of each of the first cameras and an internal parameter matrix of each of the first cameras. Similar to the first camera data, the second camera data may be a data set, and the second camera data may include an extrinsic parameter matrix of each second camera and an internal parameter matrix of each second camera and other parameters. It should be noted that the first camera data may also include original images captured by each of the first cameras, and the second camera data may also include real images captured by each of the second cameras.

Exemplarily, the first vehicle may be an old model vehicle, the second vehicle may be a new model vehicle, and both the first camera data and the second camera data may be stored in a corresponding database.

Wherein, the external parameter matrix of the target camera and the internal parameter matrix of the target camera may be obtained simultaneously or sequentially. The external parameter matrix may include rotation parameters and translation parameters converted from the world coordinate system to the camera coordinate system, for converting the coordinate points of the world coordinate system to the coordinate points of the camera coordinate system; the internal parameter matrix may include the camera itself parameters, such as the focal length of the camera, etc., the internal parameter matrix can be used to convert the coordinate points of the camera coordinate system to the coordinate points of the pixel coordinate system. The internal parameter matrix is usually fixed, while the external parameter matrix can be related to the camera's position, orientation and other parameters.

Fig. 2 shows a schematic diagram of a target position and a target camera according to an embodiment of the present application.

As shown in FIG. 2 , multiple first cameras may be installed on the first vehicle, and multiple second cameras may be installed on the second vehicle. Among the multiple first cameras, one of the multiple first cameras can be used as the target camera, the target camera exists on the first vehicle, and the same position as the target camera on the second vehicle and There is no corresponding camera.

Specifically, the position of the target camera is the same as a preset target position on the second vehicle, and the target position is different from the positions of the second cameras. In the embodiment of the present application, there is no camera at the same position as the target camera of the first vehicle in the second vehicle. Therefore, several second cameras adjacent to the target position can be used to simulate the simulated images that may be taken when the second camera is installed at the target position.

Because the position of the first camera installed on the first vehicle is different from that of the second camera installed on the second vehicle, or the model of the first camera installed on the first vehicle is different from the model of the second camera installed on the second vehicle different, resulting in a difference between the angle of view of the first camera installed on the first vehicle and the angle of view of the second camera installed on the second vehicle, so the camera data of the first vehicle cannot be directly used for the second vehicle, and the method of this application needs to be adopted A method transforms camera data from a first vehicle for a second vehicle. In the case where the position of the second camera is the same as that of the first camera, the first camera data corresponding to the first camera may be directly transplanted to the second camera at the same position as the first camera.

Step S2: Determine at least one second camera adjacent to the target position according to the extrinsic parameter matrix of the target camera and the extrinsic parameter matrix of each of the second cameras;

In the embodiment of the present application, since there is no camera at the same position as the target camera in the second vehicle, it is necessary to use at least one second camera near the target position of the second vehicle to simulate the simulation image of the target position, so that For the image captured after assuming a camera is installed at the target location. It is worth noting that since the external parameter matrix reflects the angle of view information of the camera, the process of determining at least one second camera adjacent to the target position does not require the participation of the internal parameter matrix, thereby improving the at least one second camera. Selected efficiency of the second camera.

Further, according to the extrinsic parameter matrix of the target camera and the extrinsic parameter matrix of each of the second cameras, determining at least one second camera adjacent to the target position may include:

Step S21: extracting a first translation parameter, a second translation parameter and a third translation parameter corresponding to the target camera through the external parameter matrix of the target camera;

For example, N first cameras C1, C2...C _N are installed on the first vehicle, where N is a positive integer. For each first camera, an extrinsic parameter matrix corresponding to the first camera can be obtained. Exemplarily, for the target camera C1, the first translation parameter, the second translation parameter and the third translation parameter in the external parameter matrix corresponding to the target camera can be expressed as t _x(C1) , _ty(C1) , t _z(C1) represents the translation parameter of the target object captured by the target camera from the world coordinate system to the camera coordinate system.

Step S22: extracting the fourth translation parameter, the fifth translation parameter and the sixth translation parameter corresponding to each second camera through the external parameter matrix of each second camera;

Wherein, the fourth translation parameter, the fifth translation parameter and the sixth translation parameter may respectively be t _x , _ty , t _z in the extrinsic parameter matrix of each of the second cameras, representing the The translation parameter of the target object from the world coordinate system to the camera coordinate system.

Step S23: Based on the fourth translation parameter, the fifth translation parameter and the sixth translation parameter corresponding to each of the second cameras and the first translation parameter, the second translation parameter and the third translation parameter corresponding to the target camera, respectively calculating the Euclidean distance between each of the second cameras and the target camera to obtain a plurality of camera distances corresponding to the plurality of second cameras;

Exemplarily, for the target camera C1 on the first vehicle and any second camera on the second vehicle, the translation parameters corresponding to these two cameras can be subjected to square root calculation to obtain the Euclidean distance corresponding to the second camera. Specifically, the first translation parameter can be subtracted from the fourth translation parameter and squared, the second translation parameter can be subtracted from the fifth translation parameter, and the second translation parameter can be subtracted from the fifth translation parameter, and the third translation parameter can be subtracted from the sixth translation parameter. square, and the root sign of the total sum result can be used to obtain the camera distance corresponding to the second camera. Similarly, the camera distances corresponding to the other second cameras can be obtained.

Step S24: Selecting at least one second camera that is lower than a preset camera distance threshold as at least one second camera adjacent to the target position.

Wherein, the camera distance threshold can be set as required. The at least one second camera may be n second cameras, where n is an integer. Optionally, n may be set to 2 or 3, and a 2-way second camera or a 3-way second camera close to the target camera is selected as a reference.

By using the external parameter matrix of the target camera and the external parameter matrix of each of the second cameras to determine at least one second camera that is close to the target position, the embodiment of the present application can improve the accuracy of camera data migration of different vehicles. performance while reducing the amount of calculated data.

Step S3: Based on the real image corresponding to each of the at least one second camera, the depth map, and the first camera data corresponding to the target camera, a simulation image matching the target position is formed;

Further, before forming a simulated image matching the target position based on the real image corresponding to each of the at least one second camera, the depth map, and the first camera data corresponding to the target camera, the target position Simulation image re-injection methods include:

Step S301: Process the real images captured by the at least one second camera based on the SFM algorithm, and generate depth maps corresponding to the at least one second camera.

Wherein, the second camera data may be a data set, including a plurality of real images captured by each second camera. In step S301, the at least one second camera may be found in the second camera data through the number of the second camera, and then multiple real images captured by the at least one second camera are directly extracted.

Wherein, the depth map can be used to represent the distance between the target object captured by the target camera and the target camera, that is, the target depth information. For example, for the second vehicle, a target camera is installed on the left front of the second vehicle, the traffic lights photographed by the target camera are 10 meters away from the straight-line distance of the camera, and the straight-line distance of the pedestrians photographed by the target camera from the target camera is 15m, at this time traffic lights and pedestrians can be used as the target objects. There may be one or more target objects, and the distance between the target object and the second vehicle camera may be specifically measured based on the geometric center of the target object and the optical center of the camera.

In practical applications, the coordinates of the target object captured by the target camera can be calibrated using the world coordinate system. The origin of the world coordinate system has nothing to do with the specific position of the target camera, and can be selected according to actual needs. Usually, the coordinates of the target object in the world coordinate system cannot be directly projected to the image of the two-dimensional plane, and further transformation is required. For example, the coordinates in the world coordinate system can be transformed into the camera coordinate system by using the external parameter matrix, and then the corresponding coordinates in the camera coordinate system can be transformed into the pixel coordinate system by using the internal parameter matrix. Exemplarily, the origin of the camera coordinate system may be the optical center of the camera, and the origin of the pixel coordinate system may be located at the upper left corner of the captured image.

Wherein, the depth information can be obtained through radar detection, and can also be obtained through binocular vision. It can be understood that there are many ways to acquire the depth information, which are not limited in this application.

Among them, the SFM algorithm is also called the Structure From Motion algorithm, which can reconstruct a three-dimensional structure from a series of multiple two-dimensional image sequences containing visual motion information.

Exemplarily, based on the SFM algorithm, it is possible to select two adjacent real images in the multiple real images for calculation, then match the feature points in different real images, and then calculate the corresponding fundamental matrix and intrinsic matrix, and then Depth maps corresponding to the at least one second camera are reconstructed according to the fundamental matrix and the intrinsic matrix. Each second camera in the at least one second camera corresponds to a depth map. For any second camera in the at least one second camera, the depth map corresponding to the second camera may include the distance between the target object and the second camera.

Further, based on the real image corresponding to each of the at least one second camera, the depth map, and the first camera data corresponding to the target camera, forming a simulation image matching the target position includes:

Step S31: Obtain the internal parameter matrix of each second camera in the at least one second camera;

Wherein, in step S31, the internal parameter matrix of each second camera in the at least one second camera may be extracted from the second camera data.

Step S32: Based on the external parameter matrix of each of the second cameras in the at least one second camera, the internal parameter matrix of each of the second cameras, and the corresponding depth map and real image of the at least one second camera, Projecting the pixels of the real image from the pixel coordinate system to the world coordinate system to obtain a plurality of first pixel points in the world coordinate system;

Wherein, the target object may include a plurality of feature points, and each feature point may correspond to a pixel point on an image captured based on the target object. Each feature point has a first pixel point in the world coordinate system. In practical applications, the feature points corresponding to each of the depth maps may be translated to the world coordinate system.

Due to the difference in the angle of view between the first camera and the target camera, and the pixel points in the pixel coordinate system cannot be directly transformed into coordinate points in the world coordinate system, each of the depth maps is translated to the world coordinate system, so as to pass through each of the depth maps. Realize the transformation process of projecting the pixel points of the plurality of real images from the pixel coordinate system to the world coordinate system. In other words, without a depth map, the transformation of the pixel points of the multiple real images projected from the pixel coordinate system to the world coordinate system cannot be realized; In the case of the external parameter matrix of the two cameras, the internal parameter matrix of each of the second cameras, and the corresponding depth maps of the at least one second camera, the pixels of the multiple real images can be projected from the pixel coordinate system to world coordinate system.

Step S33: Based on the plurality of first pixels and first camera data corresponding to the target camera, a simulation image matching the target position is formed.

Further, based on the plurality of first pixels and the first camera data corresponding to the target camera, forming a simulation image matching the target position includes:

Step S331: using the extrinsic parameter matrix of the target camera to generate the extrinsic parameter matrix of the target position;

In an example, the extrinsic parameter matrix of the target camera may be directly used as the extrinsic parameter matrix of the target position.

Step S332: According to the external parameter matrix of the target position, project some or all of the first pixel points to the camera coordinate system corresponding to the target camera to obtain a plurality of second pixel points in the camera coordinate system;

In an example, the conversion relationship between the first pixel point and the second pixel point can be expressed by formula (1) as follows:

Among them, X, Y, and Z respectively represent the first pixel of the target object in the world coordinate system; X _c , Y _c , and Z _c represent the second pixel of the target object in the camera coordinate system; the external parameter matrix The r ₁₁ -r ₃₃ in the total of 9 parameters represent the rotation parameters of the target object from the world coordinate system to the camera coordinate system; t _x , _ty , t _z in the external parameter matrix respectively represent the target object from the world coordinate system The translation parameter to transform to the camera coordinate system.

Since the origin of the world coordinate system does not coincide with the origin of the camera coordinate system, when a point in the world coordinate system needs to be projected onto the image plane, it is first necessary to use the external parameter matrix to convert the world coordinate system to the camera coordinate system. The external parameter matrix represents the conversion Rotation and translation in the process. Of course, since the target object may include multiple feature points, the actual processing object of formula (1) may be the feature points of the target object. From the perspective of the pixel coordinate system, the feature points may be individual pixel points in the captured image.

Step S333: Based on the plurality of second pixel points and the first camera data corresponding to the target camera, form a simulation image matching the target position.

Wherein, the transformation process of formula (1) can be regarded as a part of the forward transformation process, and the process of step S32 can be called an inverse transformation process.

It should be noted that in this application, three coordinate systems, namely, the world coordinate system, the camera coordinate system and the pixel coordinate system, are mainly used to calibrate the positions of the target object and the target camera. Those skilled in the art can understand that there are other possible deformations in the coordinate transformation, and the present application does not limit the deformation transformation between the various coordinate systems.

Further, according to the external parameter matrix of the target position, project part or all of the first pixel points to the camera coordinate system corresponding to the target camera to obtain a plurality of second pixel points in the camera coordinate system, including:

Step S3321: Obtain the viewing angle range of the target camera;

Wherein, the viewing angle range of the target camera may be a maximum viewing angle that the target camera can observe. For example, when the target camera is installed on the left front of the vehicle, the maximum viewing angle that the target camera can observe can be 120 degrees, and when the target camera is installed on the right front of the vehicle, the maximum viewing angle that the target camera can observe can be 180 degrees. Spend. The viewing angle ranges of the target cameras at different positions may be different. In other words, the maximum viewing angle that the target camera can observe may be related to the coordinates of the target camera itself.

Step S3322: Determine whether each first pixel in the plurality of first pixels is within the range of the viewing angle;

Step S3323: If the first pixel point is within the viewing angle range, project the first pixel point to the camera coordinate system corresponding to the target position; if the first pixel point is outside the viewing angle range, filter the first pixel.

It should be noted that the viewing angle range of the target camera may be related to factors such as the installation position of the camera itself and the performance of the camera, but the viewing angle range of the target camera is limited. For example, the viewing angle range of the target camera may capture a range of 120 degrees in the horizontal direction and 120 degrees in the vertical direction. Therefore, when projecting each first pixel point, it is necessary to determine whether each first pixel point among the plurality of first pixel points is located within the viewing angle range.

Wherein, the target object can be selected according to needs. The target object corresponding to the first image captured by the first camera is the same as the target object corresponding to the real image captured by the second camera. Both the first camera and the second camera can shoot the same target object from different angles of view.

Wherein, when the first pixel point is located within the viewing angle range, the first pixel point can be directly projected to the camera coordinate system corresponding to the target position; If the viewing angle is outside the range, the first pixel outside the viewing angle may be filtered out. By judging whether each first pixel of the plurality of first pixels is located within the viewing angle range, the embodiment of the present application can reduce the amount of coordinate data in the coordinate projection process and improve the generation efficiency of the simulation graphics.

Further, the first camera data also includes an internal parameter matrix of each of the first cameras, and based on the plurality of second pixels and the first camera data corresponding to the target camera, a matrix corresponding to the target position is formed. Matching emulated images, including:

Step S3331: Obtain the internal parameter matrix of the target camera;

Wherein, in step S3331, the internal parameter matrix of the target camera may be extracted from the first camera data.

Step S3332: Using the internal parameter matrix of the target camera to generate the internal parameter matrix of the target position;

In an example, the internal parameter matrix of the target camera may be directly used as the internal parameter matrix of the target position.

Step S3333: According to the internal parameter matrix of the target position, project each of the second pixels in the plurality of second pixels to the pixel coordinate system corresponding to the target position, and obtain a plurality of third pixel points in the pixel coordinate system. pixel;

In an example, the conversion relationship between the second pixel point and the third pixel point can be expressed by formula (2) as follows:

Among them, X _c , Y _c , and Z _c respectively represent the second pixel point of the target object in the camera coordinate system; x and y represent the third pixel point of the target object in the pixel coordinate system; c _x and _cy respectively represent the pixel coordinates corresponding to the origin of the camera coordinate system on the image; f _x and f _y in the internal parameter matrix may represent the focal length of the camera.

In addition, distortion factors may also be considered during the process of converting the second pixel point to the third pixel point through the internal parameter matrix. For example, a radial distortion coefficient and a tangential distortion coefficient are added to the internal parameter matrix. Adding distortion factors can improve the problem of offset and deformation between theoretically calculated pixels and actual pixels. In practical applications, whether to add distortion factors can be determined according to actual needs, which is not limited in this application.

It should be noted that the formula (2) is based on the principle of camera projection. The transformation process of formula (2) can also be regarded as a part of the forward transformation process. In the present application, the first pixel point may be the coordinates of the target object in the real world, and the coordinates are three-dimensional; the second pixel point may be an intermediate coordinate, and the coordinates are also three-dimensional; the third pixel point The point may be the coordinates of the target object in the captured image, and the coordinates are two-dimensional.

Step S3334: Based on the plurality of third pixel points, a simulation image matching the target position is formed.

Wherein, the simulated image can be used for the second vehicle. For example, the simulation image may be input into the data processing unit of the second vehicle, so that the data processing unit uses the simulation image for training, verification or testing.

It should be noted that, in the process of processing each of the depth maps, it is assumed that the direction of the external parameters of the virtual viewing angle of the target position is consistent, considering that the external parameters r11-r33 are consistent in the virtual camera participating in the target position and the target camera , so only the translational difference of the viewing angle is considered, and there is no need to consider the rotational difference of the viewing angle. For ease of understanding, it may also be considered that the at least one second camera is a camera that is relatively close to the target position, so the orientation of each second camera in the at least one second camera is similar to the orientation of the target camera. The positions of the left, right, up and down are deviated, which leads to the deviation of the viewing angle. Therefore, it is only necessary to realize the translation during the processing of each of the depth maps.

Step S4: injecting the simulation image back into the data processing unit of the second vehicle.

Further, injecting the simulation image back into the data processing unit of the second vehicle includes:

Step S41: using a bilinear difference algorithm and/or an image restoration algorithm to fill in missing pixels in the simulation image to obtain a fitting image corresponding to the target position;

Wherein, each pixel of the simulated image may correspond to a fixed RGB value (eg, grayscale). The missing or damaged pixels can be repaired by using a bilinear difference algorithm and/or an image inpainting algorithm. Exemplarily, the simulation image may also be completed according to semantic information. The image restoration algorithm can be performed based on Generative Adversarial Networks (GAN).

It should be noted that the use of the bilinear difference algorithm and the image repair algorithm can be performed simultaneously, or alternatively. Those skilled in the art can understand that the bilinear difference algorithm and the image restoration algorithm have multiple implementation forms, and the specific implementation of the bilinear difference algorithm and the image restoration algorithm is not discussed in this application. Not limited.

Step S42: injecting the fitting image back into the data processing unit of the second vehicle.

Wherein, there may be one or more than one fitting image. In the case that there are multiple fitting images, further fusion may be performed on the fitting images, so that the fitting image at the target position is closer to the actual situation.

Further, injecting the fitting image back into the data processing unit of the second vehicle may include:

Step S421: Input the formed fitting image that matches the target camera to the data processing unit of the second vehicle to perform neural network training to obtain a training value corresponding to the target position;

Exemplarily, the second vehicle may include an industrial computer and injection equipment. The fitting image can be decoded as video data by an industrial computer (also called a real-time machine), and then injected into the data processing unit by an injection device (such as a video injection board). The algorithm of the data processing unit may be based on a neural network model, and the neural network model uses the fitting image as an input to perform neural network training, and then obtain a training value corresponding to the target position.

Step S422: Comparing the training value with the real value collected by the camera installed at the target position to obtain a comparison result corresponding to the training value;

In an example, the training value may be compared with a real value collected by a camera installed at the target location, and the comparison result may be equal or not equal. If the comparison results are equal, it means that the fitting image can better fit the image actually collected by the camera installed at the target position; If the combined image cannot better fit the image actually collected by the camera installed at the target position, the neural network model needs to be readjusted.

Step S423: Adjust the neural network model in the data processing unit according to the comparison result.

Wherein, the neural network model may include network models such as DNN, CNN, LSTM, ResNet, etc., and the present application does not limit the type of the neural network model.

In one embodiment, since there are usually many overlapped parts between the simulated image (or fitted image) and the real image, the simulated image (or fitted image) and the real image collected by each second camera can be compared based on the overlapped part. The images are stitched together as an image to be verified. At the same time, each real image may have many overlapping parts. Therefore, based on the overlapping parts, the real images collected by each second camera can be stitched together as a reference image; then compare The similarity between the image to be verified and the reference image is obtained to obtain similarity evaluation information for representing the similarity. Exemplarily, if the similarity evaluation information is positively correlated with the similarity, the simulation image can be re-injected when the similarity evaluation information is higher than the similarity threshold, otherwise no re-injection is performed; if the similarity evaluation information is positively correlated with the similarity Negative correlation, when the similarity evaluation information is lower than the similarity threshold, the simulation image will be re-injected, otherwise no re-injection will be performed. In addition, simulated images (or fitted images) can be back-injected synchronously with other real images.

Through the above scheme of judging whether to re-inject the simulation image or the fitting image based on the similarity, the embodiment of the present application can avoid re-injecting the image with a poor simulation or fitting result (for example, poor accuracy), and solve the problem caused by re-injection. It may lead to the mismatch of images observed by the data processing unit for synchronous re-injection, thereby improving the training, verification, and testing effects of the algorithm.

It should be noted that the above solution can be mainly used in situations where there is no need to fill missing pixels in the simulation image, and the application to situations with missing pixels is not excluded.

In one example, the situation that there are pixels that need to be filled and the situation that there are no missing pixels that need to be filled can also be considered. At this time, since the difference between images when there is a deletion may be caused by the defect of the algorithm used for filling or other related defects, different similarity thresholds may also be used for different situations. For example, in the case of missing pixels, if the missing pixels need to be filled (that is, the image to be injected is a fitted image), the similarity threshold can be determined to be the first similarity threshold, and if there are no missing pixels that need to be filled (that is, the image to be injected is a simulated image), then the similarity threshold may be determined to be the second similarity threshold. Further, if the similarity evaluation information is positively correlated with the similarity, the first similarity threshold is smaller than the second similarity threshold, and if the similarity evaluation information is negatively correlated with the similarity, the first similarity threshold is greater than the second similarity threshold threshold.

To sum up, this application uses the coordinate mapping relationship and the depth map to fit the simulation image of the target position angle of view through the adaptation and adjustment of the first camera data and the second camera data, so that the first camera data can be adapted to the For the second vehicle, further enrich the second camera data of the second vehicle, enhance the adaptability of the second camera data, and expand the application range of the second camera data. In addition, the adapted simulation images can be used for neural network training, thereby improving the training accuracy of the neural network. At the same time, based on the first camera data and the second camera data, new models can be quickly adapted, which also improves the efficiency of new model development and testing. .

Fig. 3 shows a block diagram of a device for reinjecting a simulated image of a target camera according to an embodiment of the present application.

As shown in FIG. 3 , the simulated image reinjection device 30 of the target camera in the embodiment of the present application may include:

The camera data obtaining module 31 is used to obtain the first camera data of the first camera set of the first vehicle and the second camera data of the second camera set of the second vehicle, the first camera data includes the exterior of each first camera A parameter matrix, the first camera set includes a preset target camera, the second camera set includes a plurality of second cameras, the position of the target camera is the same as the preset target position on the second vehicle, and The target position is different from the position of each of the second cameras, and the second camera data includes an extrinsic parameter matrix of the second camera and a real image captured by the second camera;

A camera determination module 32, configured to determine at least one second camera adjacent to the target position according to the extrinsic parameter matrix of the target camera and the extrinsic parameter matrix of each of the second cameras;

An image forming module 33, configured to form a simulation image matching the target position based on the real images and depth maps corresponding to the at least one second camera, and the first camera data corresponding to the target camera;

The re-injection module 34 is configured to re-inject the simulation image into the data processing unit of the second vehicle.

In addition, the present application provides a computer-readable medium, on which a computer program is stored, and when the computer program is executed by a processor, the simulation image reinjection method for the target position is realized.

Further, the present application also provides an electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are processed by the one or more When executed by the processor, the one or more processors implement the simulated image reinjection method of the target position.

Fig. 4 shows a schematic structural diagram of an electronic device according to an embodiment of the present application.

As shown in FIG. 4 , the electronic device can be used to realize the simulated image reinjection method of the target position. Specifically, the electronic device may include a computer system. It should be noted that the electronic device shown in FIG. 3 is only an example, and should not limit the functions and scope of use of this embodiment of the present application.

As shown in FIG. 3 , the computer system includes a central processing unit (Central Processing Unit, CPU) 1801, which can be randomly accessed according to a program stored in a read-only memory (Read-Only Memory, ROM) 1802 or loaded from a storage part 1808. Various appropriate actions and processes are executed by programs in the RAM (Random Access Memory, RAM) 1803, for example, the methods described in the above-mentioned embodiments are executed. In the RAM 1803, various programs and data necessary for system operation are also stored. The CPU 1801 , ROM 1802 , and RAM 1803 are connected to each other via a bus 1804 . An input/output (Input/Output, I/O) interface 1805 is also connected to the bus 1804 .

The following components are connected to the I/O interface 1805: an input section 1806 including a keyboard, a mouse, etc.; an output section 1807 including a cathode ray tube (Cathode Ray Tube, CRT), a liquid crystal display (Liquid Crystal Display, LCD), etc., and a speaker ; a storage section 1808 including a hard disk or the like; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, a modem, or the like. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to the I/O interface 1805 as needed. A removable medium 1811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1810 as necessary so that a computer program read therefrom is installed into the storage section 1808 as necessary.

In particular, according to the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, where the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via communication portion 1809 and/or installed from removable media 1811 . When this computer program is executed by a central processing unit (CPU) 1801, various functions defined in the system of the present application are performed.

It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable Read-Only Memory (Erasable Programmable Read Only Memory, EPROM), flash memory, optical fiber, portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage device, magnetic storage device, or any suitable The combination. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which a computer-readable computer program is carried. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. . A computer program embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Wherein, each block in the flowchart or block diagram may represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or part of the code includes one or more executable instruction. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block in the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a A combination of dedicated hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software or by hardware, and the described units may also be set in a processor. Wherein, the names of these units do not constitute a limitation of the unit itself under certain circumstances.

As another aspect, the present application also provides a computer-readable medium. The computer-readable medium may be included in the electronic device described in the above-mentioned embodiments; or it may exist independently without being assembled into the electronic device. middle. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by an electronic device, the electronic device is made to implement the methods described in the above-mentioned embodiments.

It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. Actually, according to the embodiment of the present application, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided to be embodied by a plurality of modules or units.

Through the description of the above implementations, those skilled in the art can easily understand that the example implementations described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be CD-ROM, U disk, mobile hard disk, etc.) or on the network , including several instructions to make a computing device (which may be a personal computer, server, touch terminal, or network device, etc.) execute the method according to the embodiment of the present application.

In the foregoing embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

The simulation image reinjection method of the target position provided by the embodiment of the present application and its related equipment have been introduced in detail above. In this paper, specific examples are used to illustrate the principle and implementation of the present application. The description of the above embodiment is only used To help understand the technical solutions and core ideas of the present application; those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these The modification or replacement does not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.

Claims (10)

1. A method for artificial image reinjection of a target location, the method comprising:

acquiring first camera data of a first camera set of a first vehicle and second camera data of a second camera set of a second vehicle, wherein the first camera data comprise an external parameter matrix of each first camera, the first camera set comprises a preset target camera, the second camera set comprises a plurality of second cameras, the position of the target camera is the same as a preset target position on the second vehicle, the target position is different from the position of each second camera, and the second camera data comprise the external parameter matrix of the second camera and a real image shot by the second camera;

determining at least one path of second camera adjacent to the target position according to the external parameter matrix of the target camera and the external parameter matrix of each second camera;

forming a simulation image matched with the target position based on the real image and the depth map corresponding to the at least one path of second camera and the first camera data corresponding to the target camera;

reinjecting the simulated image to a data processing unit of the second vehicle.

2. The method of claim 1, wherein determining at least one path of second cameras adjacent to the target location according to the extrinsic parameter matrix of the target camera and the extrinsic parameter matrix of each second camera comprises:

extracting a first translation parameter, a second translation parameter and a third translation parameter corresponding to the target camera through the external parameter matrix of the target camera;

respectively extracting a fourth translation parameter, a fifth translation parameter and a sixth translation parameter corresponding to each second camera through an external parameter matrix of each second camera;

respectively calculating Euclidean distances between the second cameras and the target camera based on a fourth translation parameter, a fifth translation parameter and a sixth translation parameter corresponding to the second cameras and a first translation parameter, a second translation parameter and a third translation parameter corresponding to the target camera to obtain a plurality of camera distances corresponding to the second cameras;

and selecting at least one path of second camera lower than a preset camera distance threshold value as at least one path of second camera adjacent to the target position.

3. The method according to claim 1, wherein before forming the simulated image matching the target position based on the real image, the depth map and the first camera data corresponding to the target camera, the simulated image reinjection method comprises:

and processing the real image shot by the at least one second camera based on an SFM algorithm to generate a depth map corresponding to the at least one second camera.

4. The method of claim 1, wherein forming a simulated image matching the target position based on the real image, the depth map and the first camera data corresponding to the target camera, the real image and the depth map corresponding to each of the at least one second camera, comprises:

acquiring an internal parameter matrix of each second camera in the at least one path of second cameras;

based on the external parameter matrix of each second camera in the at least one path of second cameras, the internal parameter matrix of each second camera, the depth map and the real image corresponding to the at least one path of second cameras, projecting the pixel points of the real image from the pixel coordinate system to the world coordinate system to obtain a plurality of first pixel points under the world coordinate system;

and forming a simulated image matched with the target position based on the plurality of first pixel points and the first camera data corresponding to the target camera.

5. The method of claim 4, wherein forming a simulated image matching the target location based on the plurality of first pixel points and first camera data corresponding to the target camera comprises:

generating an extrinsic parameter matrix of the target location using an extrinsic parameter matrix of the target camera;

according to the external parameter matrix of the target position, projecting part or all of the first pixel points to a camera coordinate system corresponding to the target camera to obtain a plurality of second pixel points under the camera coordinate system;

and forming a simulation image matched with the target position based on the plurality of second pixel points and the first camera data corresponding to the target camera.

6. The method of claim 5, wherein the step of projecting part or all of the first pixel points to a camera coordinate system corresponding to the target camera according to the extrinsic parameter matrix of the target position to obtain a plurality of second pixel points in the camera coordinate system comprises:

acquiring a visual angle range of the target camera;

judging whether each first pixel point in the plurality of first pixel points is located in the visual angle range or not;

if the first pixel point is located in the visual angle range, projecting the first pixel point to a camera coordinate system corresponding to the target position; and if the first pixel point is positioned outside the visual angle range, filtering the first pixel point.

7. The method of claim 5, wherein the first camera data further includes an internal parameter matrix of each first camera, and the forming of the simulated image matching the target position based on the plurality of second pixel points and the first camera data corresponding to the target camera comprises:

acquiring an internal parameter matrix of the target camera;

generating an internal parameter matrix of the target position using the internal parameter matrix of the target camera;

according to the internal parameter matrix of the target position, projecting each second pixel point of the second pixel points to a pixel coordinate system corresponding to the target position to obtain a plurality of third pixel points under the pixel coordinate system;

and forming a simulation image matched with the target position based on the third pixel points.

8. The method of claim 1, wherein the reinjecting the simulated image to the data processing unit of the second vehicle comprises:

filling missing pixel points in the simulation image by adopting a bilinear difference algorithm and/or an image restoration algorithm to obtain a fitting image corresponding to the target position;

reinjecting the fitted image to a data processing unit of the second vehicle.

9. An apparatus for annotating an emulated image of a target location, said apparatus comprising:

the camera data acquisition module is used for acquiring first camera data of a first camera set of a first vehicle and second camera data of a second camera set of a second vehicle, wherein the first camera data comprise an external parameter matrix of each first camera, the first camera set comprises a preset target camera, the second camera set comprises a plurality of second cameras, the position of the target camera is the same as a preset target position on the second vehicle, the target position is different from the position of each second camera, and the second camera data comprise an external parameter matrix of the second camera and a real image shot by the second camera;

the camera determining module is used for determining at least one path of second camera adjacent to the target position according to the external parameter matrix of the target camera and the external parameter matrix of each second camera;

the image forming module is used for forming a simulation image matched with the target position based on a real image and a depth map which are respectively corresponding to the at least one path of second camera and first camera data corresponding to the target camera;

and the reinjection module is used for reinjecting the simulation image to a data processing unit of the second vehicle.

10. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the simulated signal triggering method for vehicle testing as claimed in any of claims 1 to 8.

Images