CN114757301A - Vehicle-mounted visual perception method and device, readable storage medium and electronic equipment - Google Patents

Abstract

The embodiment of the disclosure discloses a vehicle-mounted visual perception method and device, a readable storage medium and electronic equipment, wherein the method comprises the following steps: acquiring images at a plurality of continuous moments by a plurality of vehicle-mounted cameras arranged at preset positions on a vehicle to obtain a plurality of image frame sets; each image frame set comprises a plurality of image frames collected by the same vehicle-mounted camera, and each vehicle-mounted camera corresponds to one image frame at each moment; performing feature extraction on the image frames in the plurality of image frame sets through a coding branch network in a first network model to obtain a plurality of first feature maps; performing space fusion and time sequence fusion on the plurality of first feature maps through a fusion branch network in the first network model to obtain a second feature map under a bird's-eye view image coordinate system; and identifying the second feature map based on a network model corresponding to a preset perception task, and determining a perception result corresponding to the preset perception task.

Description

In-vehicle visual perception method and device, readable storage medium, electronic device

technical field

The present disclosure relates to the technical field of automatic driving, and in particular, to a vehicle visual perception method and device, a readable storage medium, and an electronic device.

Background technique

In the field of autonomous driving, the perception results of multiple in-vehicle cameras in their respective coordinate systems in the off-vehicle perception system cannot be directly used for subsequent prediction and regulation systems. The unified expression is expressed in the self-vehicle coordinate system; in the prior art, the fusion of different perspectives is usually performed separately through post-processing.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned technical problems, the present disclosure is made. Embodiments of the present disclosure provide a vehicle-mounted visual perception method and apparatus, a readable storage medium, and an electronic device.

According to an aspect of the embodiments of the present disclosure, a vehicle-mounted visual perception method is provided, including:

A plurality of vehicle-mounted cameras set at preset positions on the vehicle are used to collect images at multiple times in a row, so as to obtain multiple image frame sets; wherein, each of the image frame sets includes multiple frames collected based on the same vehicle-mounted camera. Image frames, each of the on-board cameras corresponds to one image frame at each moment;

Perform feature extraction on the image frames included in the plurality of image frame sets through the coding branch network in the first network model to obtain a plurality of first feature maps;

Perform spatial fusion and time series fusion on the plurality of first feature maps through the fusion branch network in the first network model to obtain a second feature map in the bird's-eye image coordinate system;

The second feature map is identified based on the network model corresponding to the preset sensing task, and the sensing result corresponding to the preset sensing task is determined.

According to another aspect of the embodiments of the present disclosure, a vehicle-mounted visual perception device is provided, including:

The image acquisition module is used for acquiring multiple image frame sets through multiple vehicle-mounted cameras set at preset positions on the vehicle at multiple times in a row, and each set of image frames includes a set of images based on the same set of image frames. Multiple frames of image frames collected by the vehicle-mounted camera, each of the image frames corresponds to a moment;

an encoding module, configured to perform feature extraction on the image frames included in the multiple image frame sets obtained by the image acquisition module through the encoding branch network in the first network model to obtain multiple first feature maps;

a fusion module, configured to perform spatial fusion and time series fusion on a plurality of first feature maps determined by the coding module through a fusion branch network in the first network model, to obtain a second feature map in the bird's-eye image coordinate system;

A perception module, configured to identify the second feature map determined by the fusion module based on a network model corresponding to a preset perception task, to obtain a perception result corresponding to the preset perception task.

According to yet another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, where the storage medium stores a computer program, and the computer program is used to execute the vehicle-mounted visual perception method described in any of the foregoing embodiments.

According to yet another aspect of the embodiments of the present disclosure, there is provided an electronic device, the electronic device comprising:

processor;

a memory for storing the processor-executable instructions;

The processor is configured to read the executable instructions from the memory, and execute the instructions to implement the vehicle visual perception method according to any one of the foregoing embodiments.

Based on the in-vehicle visual perception method and device, readable storage medium, and electronic device provided by the above-mentioned embodiments of the present disclosure, the end-to-end learning of the neural network is realized by realizing spatial fusion and time sequence fusion within the first neural network. The use of post-processing fusion can effectively avoid the complexity of spatial fusion and time-series fusion of images during post-processing, and avoid misidentifying the same target as multiple targets in post-processing.

The technical solutions of the present disclosure will be further described in detail below through the accompanying drawings and embodiments.

Description of drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification, and are used to explain the present disclosure together with the embodiments of the present disclosure, and do not limit the present disclosure. In the drawings, the same reference numbers generally refer to the same components or steps.

FIG. 1 is a schematic structural diagram of a first network model provided by an exemplary embodiment of the present disclosure.

FIG. 2 is a schematic flowchart of a vehicle-mounted visual perception method provided by an exemplary embodiment of the present disclosure.

FIG. 3 is a schematic flowchart of step 203 in the embodiment shown in FIG. 2 of the present disclosure.

FIG. 4 is a schematic flowchart of step 2031 in the embodiment shown in FIG. 3 of the present disclosure.

FIG. 5a is a schematic flowchart of step 401 in the embodiment shown in FIG. 4 of the present disclosure.

Fig. 5b is a schematic diagram of a conversion relationship between a bird's-eye view imaging plane and a preset plane of the ego vehicle coordinate system in an exemplary embodiment of the present disclosure.

FIG. 6 is a schematic flowchart of step 2032 in the embodiment shown in FIG. 3 of the present disclosure.

FIG. 7 is a schematic flowchart of step 204 in the embodiment shown in FIG. 2 of the present disclosure.

FIG. 8 is a schematic structural diagram of a vehicle-mounted visual perception device provided by an exemplary embodiment of the present disclosure.

FIG. 9 is a schematic structural diagram of a vehicle-mounted visual perception device provided by another exemplary embodiment of the present disclosure.

FIG. 10 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure.

Detailed ways

Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present disclosure, not all of the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited by the example embodiments described herein.

It should be noted that the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

Those skilled in the art can understand that terms such as "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices, or modules, etc., and neither represent any specific technical meaning, nor represent any difference between them. the necessary logical order of .

It should also be understood that, in the embodiments of the present disclosure, "a plurality" may refer to two or more, and "at least one" may refer to one, two or more.

It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more in the case of no explicit definition or contrary indications given in the context.

In addition, the term "and/or" in the present disclosure is only an association relationship to describe associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, and A and B exist at the same time , there are three cases of B alone. In addition, the character "/" in the present disclosure generally indicates that the related objects are an "or" relationship.

It should also be understood that the description of the various embodiments in the present disclosure emphasizes the differences between the various embodiments, and the same or similar points can be referred to each other, and for the sake of brevity, they will not be repeated.

Meanwhile, it should be understood that, for the convenience of description, the dimensions of various parts shown in the accompanying drawings are not drawn in an actual proportional relationship.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application or uses in any way.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further discussion in subsequent figures.

Embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal equipment, computing systems, environments and/or configurations suitable for use with terminal equipment, computer systems, servers, etc. electronic equipment include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients computer, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the foregoing, among others.

Electronic devices such as terminal devices, computer systems, servers, etc., may be described in the general context of computer system-executable instructions, such as program modules, being executed by the computer system. Generally, program modules may include routines, programs, object programs, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer systems/servers may be implemented in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

Application overview

In the process of realizing the present disclosure, the inventor found that the vehicle-mounted visual perception method provided in the prior art usually adopts a post-processing method. After the neural network model outputs the perception result, the perception results of different cameras are fused through some rules. ; This method has at least the following problems: when the perception results of the overlapping areas of adjacent viewing angles are fused by post-processing, due to the overlap between multiple viewing angles, the detection results will be obtained for the same target in images from different viewing angles, and the same object will be detected. Targets appear in different positions in different images, which requires complex recognition and detection in post-processing fusion, and it is easy to identify the same target as multiple targets, which leads to ambiguity.

Exemplary network structure

FIG. 1 is a schematic structural diagram of a first network model provided by an exemplary embodiment of the present disclosure. As shown in FIG. 1 , the first network model in this embodiment includes: an encoding branch network 101 , a fusion branch network 102 and a decoding branch network 103 .

In this embodiment, feature extraction is performed on the image frames collected by multiple vehicle-mounted cameras through the encoding branch network 101 to obtain multiple first feature maps; wherein, the multiple vehicle-mounted cameras are vehicle-mounted cameras arranged at different preset positions on the vehicle, Each vehicle-mounted camera corresponds to a viewing angle, and the shooting field of view corresponding to multiple vehicle-mounted cameras can cover the surrounding environment of the vehicle. In order to avoid blind spots, the shooting field of view of multiple vehicle-mounted cameras may overlap. During the driving of the vehicle, each vehicle-mounted camera continuously collects multiple images at multiple times based on a viewing angle at the set position, and obtains the image frame set corresponding to each vehicle-mounted camera. It can be understood that each vehicle-mounted camera collects a frame image frames, each image frame set includes multiple frames of image frames collected by the same vehicle camera and arranged based on time series. Optionally, the encoding branch network 101 may be any network that can implement feature extraction, for example, a convolutional neural network, etc. This embodiment does not limit its specific network structure.

Through the fusion branch network 102, spatial fusion and time series fusion are performed on the plurality of first feature maps to obtain the second feature map in the bird's-eye image coordinate system; wherein, the spatial fusion combines the data collected by each vehicle-mounted camera at the same time among the multiple vehicle-mounted cameras at the same time. Multi-frame image frames are fused to obtain the third feature map in the bird's-eye image coordinate system; time series fusion is to fuse the third feature map corresponding to each time in multiple moments to obtain the second feature map after time series fusion.

The second feature map obtained by the fusion branch network 102 is decoded by the decoding branch network 103 to obtain a decoded second feature map; optionally, the decoded second feature map in this embodiment may be suitable for subsequent sensing tasks The feature map of the structure, optionally, the decoding branch network 103 may be any network that can implement feature extraction, for example, a convolutional neural network, etc. This embodiment does not limit the specific network structure.

The first network model provided in this embodiment is to implement the visual perception task, therefore, any visual perception task can be connected after the first network model, and the visual perception task may include but not limited to: segmentation task, detection task, classification task, etc.

In this embodiment, the space fusion and time fusion are realized inside the first network model, so that the disturbance of the internal and external parameters of the camera caused by the change of the camera posture during the driving process of the vehicle does not directly affect the perception result, and the disturbance is overcome by training and learning the first network model. Influence, so that the output perception results are more stable, and are not affected by the camera attitude change during the driving process of the vehicle; in the first network model internal space fusion and time fusion, there is no need to perform space fusion and time fusion in the post-processing process, avoiding the need for Fusion in post-processing reduces the complexity of post-processing, enables the first network model to achieve end-to-end learning, and has the potential of joint learning with candidate visual perception tasks. The perceptual fusion method requires less computational complexity of the chip.

Exemplary method

FIG. 2 is a schematic flowchart of a vehicle-mounted visual perception method provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to an electronic device, as shown in FIG. 2 , including the following steps:

In step 201, a plurality of vehicle-mounted cameras set at preset positions on the vehicle are used to collect images at a plurality of consecutive moments to obtain a plurality of image frame sets.

Wherein, each image frame set includes multiple image frames collected based on the same vehicle-mounted camera, and each vehicle-mounted camera corresponds to one image frame at each moment.

Optionally, since each vehicle-mounted camera corresponds to an image frame set, multiple vehicle-mounted cameras can obtain multiple image frame sets; each vehicle-mounted camera can correspond to different directions, and the image acquisition ranges corresponding to the multiple vehicle-mounted cameras can be There is overlap or there is no overlap. Usually, multiple on-board cameras are set in multiple preset positions to capture the corresponding omnidirectional angles of the vehicle, so as to provide more information for automatic driving or assisted driving.

Step 202 , perform feature extraction on the image frames included in the multiple image frame sets through the coding branch network in the first network model to obtain multiple first feature maps.

Optionally, feature extraction is performed on the image frames included in the multiple image frame sets in turn by the encoding branch network in the first network model, to obtain multiple first feature maps; for the encoding branch network in this embodiment, refer to FIG. 1 . The provided encoding branch network 101 understands and implements feature extraction on the image frame.

Step 203 , performing spatial fusion and time series fusion on the plurality of first feature maps through the fusion branch network in the first network model to obtain a second feature map in the bird's-eye image coordinate system.

Optionally, the encoding branch network in this embodiment can be understood with reference to the fusion branch network 102 provided in FIG. 1 to implement fusion of the first feature map.

Step 204: Identify the second feature map based on the network model corresponding to the preset sensing task, and determine the sensing result corresponding to the preset sensing task.

The preset perception task in this embodiment may be any visual perception task, for example, a segmentation task, a detection task, a classification task, etc. The recognition operation of this step is implemented through a network model corresponding to the visual perception task.

The vehicle visual perception method provided by the above embodiments of the present disclosure realizes the end-to-end learning of the neural network by implementing spatial fusion and time sequence fusion within the first neural network. The complexity of spatial fusion and temporal fusion of images during processing, as well as avoiding misidentification of the same target as multiple targets in post-processing.

As shown in FIG. 3 , on the basis of the above-mentioned embodiment shown in FIG. 2 , step 203 may include the following steps:

Step 2031 , for each moment in the plurality of moments, perform spatial fusion of the plurality of first feature maps corresponding to the moment to obtain a plurality of third feature maps in the bird's-eye image coordinate system.

Among them, each third feature map corresponds to a moment.

In this embodiment, the image frame collected by each vehicle-mounted camera is an image frame in the camera coordinate system corresponding to the vehicle-mounted camera. Then, the multiple feature maps in the bird's-eye image coordinate system are fused. The fusion of multiple feature maps in the same coordinate system can be achieved by the feature map fusion method in related technologies, for example, element-by-element addition, in the feature channel dimension. Splicing, element-wise maximum value, neural network fusion, etc.

Step 2032: Perform time series fusion on the plurality of third feature maps to obtain a second feature map.

In this embodiment, each third feature map corresponds to a moment, and by reconstructing the third feature map corresponding to each moment to a certain moment (for example, collecting the latest moment corresponding to the last image frame) In the feature map, the second feature map can be obtained by fusing the reconstructed third feature maps. The fusion in this step can also be realized by the feature map fusion method in the related art. addition, splicing in the feature channel dimension, taking the maximum value element by element, neural network fusion, etc. In this embodiment, spatial fusion is performed first, and then time series fusion is performed, which makes time series fusion easier to implement and accelerates the speed of feature map fusion.

As shown in FIG. 4 , on the basis of the above-mentioned embodiment shown in FIG. 3 , step 2031 may include the following steps:

Step 401 , perform homography transformation on each of the plurality of first feature maps, respectively, to obtain a plurality of transformed feature maps in the bird's-eye image coordinate system.

Among them, the homography transformation (or projective transformation) has 8 degrees of freedom and is used to describe the mapping relationship of points on two planes. Through the homography transformation, each first feature map in the image coordinate system is converted into the bird's-eye image coordinate system, and then multiple transformed feature maps can be obtained.

Step 402 , perform point-by-point fusion of the plurality of transformed feature maps to obtain a third feature map in the bird's-eye image coordinate system.

In this embodiment, multiple transformation feature maps in the same coordinate system are obtained through homography transformation, and the sizes of the multiple transformation feature maps are the same. Therefore, the fusion of multiple transformation feature maps can be realized by point-by-point fusion. The method of fusion, point-by-point fusion may include, but is not limited to: element-by-element addition, splicing in the feature channel dimension, element-by-element maximum value, neural network fusion, etc. The multiple first feature maps are mapped to the same coordinate system, the feature map of each view is converted from the camera view to the bird's-eye view, and the fusion is realized under the same coordinate system, so that the third feature map after fusion can have the vehicle Image features corresponding to all surrounding angles, when performing visual perception tasks based on the third feature map, the perception results of different vehicle cameras can be fused without post-processing, which improves the fusion efficiency of perception results and reduces the difficulty of fusion. .

As shown in FIG. 5a, on the basis of the above-mentioned embodiment shown in FIG. 4, step 401 may include the following steps:

Step 4011: Determine a first transformation matrix from the vehicle coordinate system corresponding to the first feature map to the camera coordinate system based on the internal parameter matrix and the external parameter matrix of the vehicle camera corresponding to each first feature map.

其中，r₁₁、r₁₂、r₁₃、r₂₁、r₂₂、r₂₃、r₃₁、r₃₂、r₃₃表示旋转参数，t₁、t₂、t₃表示平移参数。In this embodiment, the internal parameter matrix corresponding to each on-board camera is a known 3*3 matrix K, and the external parameter matrix corresponding to the on-board camera can be calculated by setting the preset position of the on-board camera on the vehicle. Based on the internal parameter matrix and The extrinsic parameter matrix can determine the first transformation matrix T _vcs2cam between the vehicle coordinate system and the camera coordinate system, and the first transformation matrix includes rotation parameters and translation parameters. For example, in an optional example, the first transformation matrix can represent for:

Among them, r ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ , r ₃₁ , r ₃₂ , and r ₃₃ represent rotation parameters, and t ₁ , t ₂ , and t ₃ represent translation parameters.

Step 4012: Determine a second transformation matrix from the bird's-eye image coordinate system to the preset plane of the self-vehicle coordinate system based on the sensing range on the preset plane of the vehicle's coordinate system and the zoom ratio and translation distance of the bird's-eye image coordinate system.

Optionally, as shown in Fig. 5b, d ₁ , d ₂ , d ₃ , and d ₄ indicate that the self-vehicle coordinate system is on a preset plane (xy plane, and O _vcs in the figure indicates the center of the xy plane in the self-vehicle coordinate system. point), based on which the translation distance of the ego vehicle coordinate system relative to the bird's-eye image coordinate system can be determined (d ₁ , d ₃ in Figure 5b ), assuming that the bird's-eye view imaging plane (the The center point can be O _BEV in Fig. 5b ) coincides with (or is parallel to) the xy plane of the self-vehicle coordinate system, and r represents the scaling ratio of the xy plane of the self-vehicle coordinate system to the bird's-eye view imaging plane, then the second transformation matrix can be Expressed as:

Step 4013 , based on the first transformation matrix, the second transformation matrix and the internal parameter matrix of the vehicle camera, determine a third transformation matrix from the bird's-eye image coordinate system to the image coordinate system corresponding to the vehicle camera.

Optionally, in this embodiment, the transformation matrix from the bird's-eye image coordinate system to the camera coordinate system can be derived through the first transformation matrix and the second transformation matrix, and then combined with the internal parameter matrix, the bird's-eye image coordinate system can be determined to the vehicle camera. The third transformation matrix T _bev2img between the corresponding image planes (corresponding to the image coordinate system).

Step 4014: Transform the first feature map based on the third transformation matrix to obtain a transformed feature map.

In this embodiment, the third transformation matrix is determined through the above determination, that is, the transformation relationship between the image coordinate system and the bird's-eye image coordinate system is determined. Through the transformation relationship, the position of the image can be reversely indexed according to the position of the bird's-eye image (u, v) as (u ₁ , v ₁ ) features, for example, feature mapping can be achieved by the following formula (1):

Thus obtain the transformation feature map f _n after the perspective conversion; Similarly, the first feature map corresponding to each perspective is carried out the above-mentioned transformation operation, obtain N (N is the number of the vehicle-mounted camera, can be set according to the specific scene) perspective transformation The transformed feature map {f _n }, n = 1, 2, ..., N; finally, feature fusion is performed on it to obtain the third feature map F _t at time t (one of multiple moments) after spatial fusion .

Optionally, on the basis of the foregoing embodiment, step 3013 may further include:

a1, based on the first transformation matrix, determine a fourth transformation matrix from the preset plane of the ego vehicle coordinate system to the camera coordinate system.

Optionally, it is assumed that the imaging plane of the bird's-eye view is coincident with (or parallel to) the xy plane of the vehicle coordinate system, and the xy plane is the plane where the z value of the three-dimensional point coordinate under the vehicle coordinate is equal to 0 (or equal to any same value). , the column corresponding to the z-axis (the third column) in the first transformation matrix T _vcs2cam can be simply eliminated. When parallel, the third column can be set as the default value to obtain the difference between the imaging plane of the bird's-eye image coordinate system and the camera coordinate system. The fourth transformation matrix T _{vcs_xy2cam between} , for example, in an optional example (the imaging plane of the bird's-eye image coordinate system coincides with the xy plane of the ego vehicle coordinate system), the fourth transformation matrix can be expressed as:

a2, based on the fourth transformation matrix and the second transformation matrix, determine a fifth transformation matrix from the bird's-eye image coordinate system to the camera coordinate system.

Optionally, after the fourth transformation matrix is determined, in combination with the second transformation matrix from the bird's-eye image coordinate system to the ego vehicle coordinate system, matrix multiplication can be performed by the fourth transformation matrix and the second transformation matrix to determine the bird's-eye image coordinate system to The fifth transformation matrix between camera coordinate systems, for example, the fifth transformation matrix T _bev2cam can be determined based on the following formula (2):

T _bev2cam =T _vcs-xy2cam ×T _bev2vcs-xy formula (2)

a3, based on the fifth transformation matrix and the internal parameter matrix of the vehicle camera, determine the third transformation matrix.

In this embodiment, it is determined that the number of rows and columns between the fifth transformation matrix and the internal parameter matrix does not correspond. In order to perform matrix multiplication (the internal parameter matrix is a matrix of 3*3), an interception operation can be performed on the fifth transformation matrix to intercept the The intercepted transformation matrix is obtained from the first three rows of the five transformation matrix. For example, the first three rows of the fifth transformation matrix T _bev2cam are intercepted to obtain the intercepted transformation matrix T′ _bev2cam . At this time, matrix multiplication is performed by intercepting the transformation matrix and the internal parameter matrix, Determine the third transformation matrix, for example, as shown in the following formula (3):

T _bev2img = K×T′ _bev2cam formula (3)

Through the above processing, a third transformation matrix can be obtained, and based on the third transformation matrix, the first feature map corresponding to the image coordinate system can be converted into the imaging plane under the bird's-eye image coordinate system. In this embodiment, the coordinate system relationship is converted, Determine the third transformation matrix for the transformation from the bird's-eye image coordinate system to the image coordinate system, and realize that the features in the image coordinate system can be directly transformed into the bird's-eye image coordinate system based on the third transformation matrix. In the fusion branch model of the first network model Fast spatial fusion is achieved, and the specific parameters in the coordinate system transformation are determined through the first network model learning, which improves the accuracy and speed of the transformation.

As shown in FIG. 6, on the basis of the above-mentioned embodiment shown in FIG. 3, step 2032 may include the following steps:

Step 601 , using the third feature map corresponding to the most recent time among the multiple moments as the reference feature map.

Optionally, when the multiple times include times t, t-1...t-s, the third feature map corresponding to the t-th time (the latest time) may be used as the reference feature map. Of course, this embodiment is only a specific example, In practical applications, the third feature map corresponding to any time can be used as the reference feature map, which does not affect the operation and results of time series fusion; and when the most recent time is selected as the parameter feature map, the result of time domain fusion corresponds to the most recent time. , that is, the outputted second feature corresponds to the most recent moment (for example, the current moment), which improves the real-time performance of the sensing result determined based on the second feature, that is, the real-time performance of the vehicle perspective perception method.

Step 602: Reconstruct each third feature map of the at least one third feature map, respectively, to obtain at least one fourth feature map.

In this embodiment, in order to realize that the third feature maps corresponding to multiple times can be fused, each third feature map is mapped into the feature space corresponding to the reference feature map, so as to perform feature fusion.

Step 603: Perform point-by-point fusion on the reference feature map and at least one fourth feature map to obtain a second feature map.

Optionally, the point-by-point fusion method may include, but is not limited to: element-by-element addition, splicing in the feature channel dimension, element-by-element maximum value, neural network fusion, etc. The timing fusion of the third feature map corresponding to each moment in the moment realizes the spatial fusion and timing fusion in the fusion branch network of the first network model in turn, and achieves the internal branch realization in the first network model, combining multiple perspectives. The multi-frame image frames collected at multiple times are fused into a second feature map in the bird's-eye image coordinate system, so that the first network model can learn end-to-end how to realize the fusion of the space and time series of the feature maps, without the need to follow the network model. External post-processing performs spatial and temporal fusion, reducing post-processing complexity and reducing ambiguity issues introduced by fusion in post-processing.

Optionally, on the basis of the foregoing embodiment, step 602 may further include:

b1. Determine the first feature corresponding to the first feature map corresponding to the reference feature map and the at least one third feature map based on the inter-frame motion transformation matrix corresponding to the vehicle camera at the latest time and at least one time corresponding to at least one third feature map At least one homography transformation matrix between graphs.

Optionally, since the vehicle's position may change when the vehicle-mounted camera collects images at different times, the camera position may change accordingly. The height d of the vehicle camera (for example, the vehicle front-view camera) from the xy plane of the vehicle coordinate system is known (it is an external parameter of the camera, which can be calculated and determined by the existing technology algorithm or determined by the information collected by the sensor, etc.), and any angle of view is calculated. The homography transformation matrix H between the two frames of images before and after. Assuming that the rotation from time t to time t-1 is R, and the translation is m, at this time, the homography transformation matrix H can be determined based on the following formula (4):

Among them, n=[0, 0, 1] ^T is the normal direction of the xy plane of the vehicle coordinate system; K represents the internal parameter matrix of the vehicle camera; d represents the height of the vehicle camera from the xy plane of the vehicle coordinate system.

b2. Based on the at least one homography transformation matrix and the third transformation matrix, determine at least one transformation matrix between the reference feature map and the at least one third feature map.

In this embodiment, a transformation matrix is determined for each third feature map in combination with the third transformation matrix of spatial transformation and the homography transformation matrix corresponding to each third feature map. Optionally, it can be determined based on the following formula (5) Transformation matrix T _temp :

b3. Reconstruct each third feature map in at least one third feature map based on each conversion matrix in the at least one conversion matrix, respectively, to obtain at least one fourth feature map.

In this embodiment, taking the third feature map at time t-1 as an example, the feature at the position (U _t-1 , V _t-1 ) after spatial fusion at time t-1 can be _mapped to (U _t , V _t ) of the spatial fusion features at time t to obtain the reconstructed feature F′ _{t at time t} , that is, the reconstruction of the third feature map at time t-1 is achieved through the transformation matrix, and the fourth The feature map, and so on, reconstruct the third feature map at each moment to obtain at least one fourth feature map; through the feature map reconstruction, the feature maps corresponding to different moments can be fused, and by executing Space fusion, and then perform time series fusion, reduces the number of feature maps processed by time series fusion, and reduces the difficulty of time series fusion. In addition, the third transformation matrix in spatial fusion is used to achieve time series fusion, which improves the repeated utilization of parameters. Improve the fusion efficiency.

As shown in FIG. 7 , on the basis of the above-mentioned embodiment shown in FIG. 2 , step 204 may include the following steps:

Step 2041 , decode the second feature map through the decoding branch network in the first network model to obtain a decoded second feature map.

Optionally, the decoding branch network in this embodiment may be understood with reference to the decoding branch network 103 in the embodiment provided in FIG. 1 .

Step 2042: Identify the decoded second feature map based on the second network model, and determine the perception result corresponding to the preset perception task.

In this embodiment, the second feature map can be mapped into the feature space required by the preset sensing task through the decoding branch network, so that the network model corresponding to the subsequent preset sensing task can directly process the mapped feature map, reducing the number of The other process of intermediate processing is realized, and the network model of the preset sensing task is directly connected with the first network model, which can realize the joint learning of the network model. Through the joint learning, the accuracy of the sensing result of the preset sensing task is improved.

Any of the vehicle-mounted visual perception methods provided by the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any in-vehicle visual perception method provided by the embodiments of the present disclosure may be executed by a processor, for example, the processor executes any of the in-vehicle visual perception methods mentioned in the embodiments of the present disclosure by invoking corresponding instructions stored in the memory. No further description will be given below.

Exemplary device

FIG. 8 is a schematic structural diagram of a vehicle-mounted visual perception device provided by an exemplary embodiment of the present disclosure. As shown in FIG. 8 , the device provided in this embodiment includes:

The image acquisition module 81 is configured to perform image acquisition at a plurality of consecutive moments through a plurality of vehicle-mounted cameras set at preset positions on the vehicle to obtain a plurality of image frame sets.

Wherein, each image frame set includes multiple frames of image frames collected based on the same vehicle-mounted camera, and each of the vehicle-mounted cameras corresponds to one frame of image frame at each moment.

The encoding module 82 is configured to perform feature extraction on the image frames included in the multiple image frame sets obtained by the image acquisition module 81 through the encoding branch network in the first network model to obtain multiple first feature maps.

The fusion module 83 is configured to perform spatial fusion and time series fusion on the plurality of first feature maps determined by the encoding module 82 through the fusion branch network in the first network model, to obtain a second feature map in the bird's-eye image coordinate system.

The sensing module 84 is configured to identify the second feature map determined by the fusion module 83 based on the network model corresponding to the preset sensing task, and obtain the sensing result corresponding to the preset sensing task.

The vehicle-mounted visual perception device provided by the above embodiments of the present disclosure realizes the end-to-end learning of the neural network by implementing spatial fusion and time sequence fusion within the first neural network. The complexity of spatial fusion and temporal fusion of images during processing, as well as avoiding misidentification of the same target as multiple targets in post-processing.

FIG. 9 is a schematic structural diagram of a vehicle-mounted visual perception device provided by another exemplary embodiment of the present disclosure. As shown in FIG. 9, in the apparatus provided by this embodiment, the fusion module 83 includes:

The spatial fusion unit 831 is configured to perform spatial fusion on the plurality of first feature maps corresponding to the moments for each moment in the plurality of moments to obtain a plurality of third feature maps in the bird's-eye image coordinate system.

Among them, each third feature map corresponds to a moment.

The time sequence fusion unit 832 is configured to perform time sequence fusion on a plurality of third feature maps to obtain a second feature map.

Optionally, the spatial fusion unit 831 is specifically configured to perform homography transformation on each of the first feature maps in the plurality of first feature maps to obtain a plurality of transformed feature maps in the bird's-eye image coordinate system; The images are fused point by point to obtain the third feature map in the bird's-eye image coordinate system.

Optionally, when the spatial fusion unit 831 performs homography transformation on each of the first feature maps of the plurality of first feature maps to obtain a plurality of transformed feature maps in the bird's-eye image coordinate system, it is used to perform the homography transformation based on each of the first feature maps. The internal parameter matrix and external parameter matrix of the vehicle camera corresponding to the feature map, determine the first transformation matrix from the vehicle coordinate system corresponding to the first feature map to the camera coordinate system; the perception range and bird's-eye view on the preset plane based on the self-vehicle coordinate system The zoom ratio and translation distance of the image coordinate system determine the second transformation matrix from the bird's-eye image coordinate system to the preset plane of the vehicle coordinate system; determine the bird's-eye image based on the first transformation matrix, the second transformation matrix and the internal parameter matrix of the vehicle camera The third transformation matrix from the coordinate system to the image coordinate system corresponding to the vehicle camera; the first feature map is transformed based on the third transformation matrix to obtain the transformed feature map.

Optionally, when the spatial fusion unit 831 determines the third transformation matrix from the bird's-eye image coordinate system to the image coordinate system corresponding to the on-board camera based on the first transformation matrix, the second transformation matrix, and the internal parameter matrix of the vehicle-mounted camera, it is used for the first transformation based on the first transformation matrix. Transformation matrix, determining the fourth transformation matrix from the preset plane of the vehicle coordinate system to the camera coordinate system; based on the fourth transformation matrix and the second transformation matrix, determining the fifth transformation matrix from the bird's-eye image coordinate system to the camera coordinate system; The fifth transformation matrix and the internal parameter matrix of the vehicle camera are used to determine the third transformation matrix.

Optionally, the time sequence fusion unit 832 is specifically configured to use the third feature map corresponding to the most recent time among the multiple moments as the reference feature map; respectively reconstruct each third feature map in at least one third feature map to obtain: at least one fourth feature map; performing point-by-point fusion of the reference feature map and the at least one fourth feature map to obtain a second feature map.

Optionally, when the time sequence fusion unit 832 reconstructs each third feature map of at least one third feature map respectively to obtain at least one fourth feature map, it is used for at least one The inter-frame motion transformation matrix corresponding to at least one moment corresponding to the feature map, determining at least one homography transformation matrix between the first feature map corresponding to the reference feature map and the first feature map corresponding to at least one third feature map; A homography transformation matrix and a third transformation matrix to determine at least one transformation matrix between the reference feature map and at least one third feature map; The third feature map is reconstructed to obtain at least one fourth feature map.

In some optional embodiments, the perception module 84 includes:

The decoding unit 841 is configured to perform decoding processing on the second feature map through the decoding branch network in the first network model to obtain a decoded second feature map.

The feature identifying unit 842 is configured to identify the decoded second feature map based on the second network model, and determine the sensing result corresponding to the preset sensing task.

Exemplary Electronics

Hereinafter, an electronic device according to an embodiment of the present disclosure will be described with reference to FIG. 10 . The electronic device can be either or both of the first device 100 and the second device 200, or a stand-alone device independent of them that can communicate with the first device and the second device to receive data from them The acquired input signal.

10 illustrates a block diagram of an electronic device according to an embodiment of the present disclosure.

As shown in FIG. 10 , the electronic device 10 includes one or more processors 11 and a memory 12 .

Processor 11 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in electronic device 10 to perform desired functions.

Memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache memory, or the like. The non-volatile memory may include, for example, read only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the vehicle-mounted visual perception method and/or the various embodiments of the present disclosure described above. Other desired features. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

In one example, the electronic device 10 may also include an input device 13 and an output device 14 interconnected by a bus system and/or other form of connection mechanism (not shown).

For example, when the electronic device is the first device 100 or the second device 200, the input device 13 may be the above-mentioned microphone or microphone array for capturing the input signal of the sound source. When the electronic device is a stand-alone device, the input device 13 may be a communication network connector for receiving the collected input signals from the first device 100 and the second device 200 .

In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.

The output device 14 can output various information to the outside, including the determined distance information, direction information, and the like. The output device 14 may include, for example, displays, speakers, printers, and communication networks and their connected remote output devices, among others.

Of course, for simplicity, only some of the components in the electronic device 10 related to the present disclosure are shown in FIG. 10 , and components such as buses, input/output interfaces, and the like are omitted. Besides, the electronic device 10 may also include any other suitable components according to the specific application.

Exemplary computer program product and computer readable storage medium

In addition to the methods and apparatus described above, embodiments of the present disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the "exemplary method" described above in this specification The steps in the in-vehicle visual perception method according to various embodiments of the present disclosure described in the section.

The computer program product may write program code for performing operations of embodiments of the present disclosure in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc. , also includes conventional procedural programming languages, such as "C" language or similar programming languages. The program code may execute entirely on the user computing device, partly on the user device, as a stand-alone software package, partly on the user computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on.

In addition, embodiments of the present disclosure may also be computer-readable storage media having computer program instructions stored thereon that, when executed by a processor, cause the processor to perform the above-described "Example Method" section of this specification The steps in the in-vehicle visual perception method according to various embodiments of the present disclosure described in .

The computer-readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses or devices, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

The basic principles of the present disclosure have been described above with reference to specific embodiments. However, it should be pointed out that the advantages, advantages, effects, etc. mentioned in the present disclosure are only examples rather than limitations, and these advantages, advantages, effects, etc. should not be considered to be A must-have for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of example and easy understanding, but not for limitation, and the above details do not limit the present disclosure to be implemented by using the above specific details.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments may be referred to each other. As for the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.

The block diagrams of devices, apparatuses, apparatuses, and systems referred to in this disclosure are merely illustrative examples and are not intended to require or imply that the connections, arrangements, or configurations must be in the manner shown in the block diagrams. As those skilled in the art will appreciate, these means, apparatuses, apparatuses, systems may be connected, arranged, configured in any manner. Words such as "including", "including", "having" and the like are open-ended words meaning "including but not limited to" and are used interchangeably therewith. As used herein, the words "or" and "and" refer to and are used interchangeably with the word "and/or" unless the context clearly dictates otherwise. As used herein, the word "such as" refers to and is used interchangeably with the phrase "such as but not limited to".

The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless specifically stated otherwise. Furthermore, in some embodiments, the present disclosure can also be implemented as programs recorded in a recording medium, the programs including machine-readable instructions for implementing methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

It should also be noted that, in the apparatus, device and method of the present disclosure, each component or each step may be decomposed and/or recombined. These disaggregations and/or recombinations should be considered equivalents of the present disclosure.

The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for the purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims (11)

1. An on-vehicle visual perception method, comprising:

acquiring images at a plurality of continuous moments by a plurality of vehicle-mounted cameras arranged at preset positions on a vehicle to obtain a plurality of image frame sets; each image frame set comprises a plurality of image frames collected by the same vehicle-mounted camera, and each vehicle-mounted camera corresponds to one image frame at each moment;

performing feature extraction on the image frames in the plurality of image frame sets through a coding branch network in a first network model to obtain a plurality of first feature maps;

performing space fusion and time sequence fusion on the plurality of first feature maps through a fusion branch network in the first network model to obtain a second feature map under a bird's-eye view image coordinate system;

and identifying the second feature map based on a network model corresponding to a preset perception task, and determining a perception result corresponding to the preset perception task.

2. The method according to claim 1, wherein the performing spatial fusion and time-series fusion on the plurality of first feature maps through a fusion branch network in the first network model to obtain a second feature map in a bird's-eye view image coordinate system comprises:

performing spatial fusion on a plurality of first feature maps corresponding to the time to obtain a plurality of third feature maps under a bird's-eye view image coordinate system for each time in the plurality of times; each third feature map corresponds to a moment;

and performing time sequence fusion on the plurality of third feature maps to obtain the second feature map.

3. The method according to claim 2, wherein the performing spatial fusion on the plurality of first feature maps corresponding to the time instants to obtain a plurality of third feature maps in a bird's-eye view image coordinate system comprises:

performing homographic transformation on each first feature map in the plurality of first feature maps respectively to obtain a plurality of transformation feature maps under a bird's-eye view image coordinate system;

and performing point-by-point fusion on the plurality of transformation characteristic graphs to obtain the third characteristic graph under the aerial view image coordinate system.

4. The method according to claim 3, wherein the performing the homographic transformation on each of the plurality of first feature maps to obtain a plurality of transformed feature maps under the bird's-eye view image coordinate system comprises:

determining a first transformation matrix from a vehicle coordinate system to a camera coordinate system corresponding to the first characteristic diagram based on an internal reference matrix and an external reference matrix of the vehicle-mounted camera corresponding to each first characteristic diagram;

determining a second transformation matrix from the aerial view image coordinate system to a preset plane of the self-vehicle coordinate system based on a sensing range on the preset plane of the self-vehicle coordinate system and the scaling and translation distance of the aerial view image coordinate system;

determining a third transformation matrix from the bird's-eye view image coordinate system to an image coordinate system corresponding to the vehicle-mounted camera based on the first transformation matrix, the second transformation matrix and an internal reference matrix of the vehicle-mounted camera;

and transforming the first characteristic diagram based on the third transformation matrix to obtain the transformation characteristic diagram.

5. The method of claim 4, wherein the determining a third transformation matrix of the bird's eye view image coordinate system to an image coordinate system corresponding to the vehicle-mounted camera based on the first transformation matrix, the second transformation matrix, and an internal reference matrix of the vehicle-mounted camera comprises:

determining a fourth transformation matrix from a preset plane of the own vehicle coordinate system to a camera coordinate system based on the first transformation matrix;

determining a fifth transformation matrix from the aerial view image coordinate system to the camera coordinate system based on the fourth transformation matrix and the second transformation matrix;

and determining the third transformation matrix based on the fifth transformation matrix and the internal reference matrix of the vehicle-mounted camera.

6. The method according to claim 4 or 5, wherein the performing time-series fusion on the plurality of third feature maps to obtain the second feature map comprises:

taking a third characteristic diagram corresponding to the latest moment in the plurality of moments as a reference characteristic diagram;

reconstructing each third feature map in the at least one third feature map respectively to obtain at least one fourth feature map;

and performing point-by-point fusion on the reference feature map and the at least one fourth feature map to obtain the second feature map.

7. The method according to claim 6, wherein the reconstructing each of the at least one third feature map to obtain at least one fourth feature map comprises:

determining at least one homography transformation matrix between the first feature diagram corresponding to the reference feature diagram and the first feature diagram corresponding to the at least one third feature diagram based on the interframe motion transformation matrix corresponding to the vehicle-mounted camera at the latest moment and the at least one moment corresponding to the at least one third feature diagram;

determining at least one transformation matrix between the reference feature map and the at least one third feature map based on the at least one homographic transformation matrix and the third transformation matrix;

and reconstructing each third characteristic diagram in the at least one third characteristic diagram respectively based on each conversion matrix in the at least one conversion matrix to obtain the at least one fourth characteristic diagram.

8. The method according to any one of claims 1 to 7, wherein the identifying the second feature map based on a second network model corresponding to a preset sensing task and determining a sensing result corresponding to the preset sensing task include:

decoding the second feature map through a decoding branch network in the first network model to obtain a decoded second feature map;

and identifying the decoded second feature map based on the second network model, and determining a perception result corresponding to the preset perception task.

9. An on-vehicle visual perception device, comprising:

the system comprises an image acquisition module, a data acquisition module and a data processing module, wherein the image acquisition module is used for acquiring images at a plurality of continuous moments through a plurality of vehicle-mounted cameras arranged at preset positions on a vehicle to obtain a plurality of image frame sets; each image frame set comprises a plurality of image frames collected by the same vehicle-mounted camera, and each vehicle-mounted camera corresponds to one image frame at each moment;

the encoding module is used for extracting the features of the image frames in the image frame sets obtained by the image acquisition module through an encoding branch network in a first network model to obtain a plurality of first feature maps;

the fusion module is used for performing space fusion and time sequence fusion on the plurality of first feature maps determined by the encoding module through a fusion branch network in the first network model to obtain a second feature map under a bird's-eye view image coordinate system;

and the perception module is used for identifying the second characteristic diagram determined by the fusion module based on a network model corresponding to a preset perception task to obtain a perception result corresponding to the preset perception task.

10. A computer-readable storage medium, storing a computer program for executing the in-vehicle visual perception method of any of the above claims 1-8.

11. An electronic device, the electronic device comprising:

a processor;

a memory for storing the processor-executable instructions;

the processor is used for reading the executable instructions from the memory and executing the instructions to realize the vehicle-mounted visual perception method of any one of the claims 1-8.

Images