CN113326820B - Driving environment sensing method and device, electronic equipment and storage medium - Google Patents
Driving environment sensing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113326820B CN113326820B CN202110880924.5A CN202110880924A CN113326820B CN 113326820 B CN113326820 B CN 113326820B CN 202110880924 A CN202110880924 A CN 202110880924A CN 113326820 B CN113326820 B CN 113326820B
- Authority
- CN
- China
- Prior art keywords
- dynamic information
- picture
- returned
- information
- dynamic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Abstract
The application relates to a driving environment sensing method, a driving environment sensing device, electronic equipment and a storage medium, wherein a dynamic information set returned by a dynamic sensing camera in a preset time period is obtained, the dynamic sensing camera is used for obtaining dynamic information of a dynamic object with changed pixel brightness, a target picture set is determined from a returned picture set returned by a camera in the preset time period according to the dynamic information set returned in the preset time period, a prediction picture set corresponding to the preset time period is determined according to the dynamic information set and the target picture set, and then the driving environment information is determined based on the returned picture set and the prediction picture set. Based on the method and the device, the dynamic information missed in the process of returning the picture by the camera according to the fixed time can be made up, and repeated extraction and analysis of the static scene by the camera can be reduced, so that the returned picture is reduced, and the operation amount of the terminal is further reduced.
Description
Technical Field
The invention relates to the technical field of automatic driving, in particular to a driving environment sensing method and device, electronic equipment and a storage medium.
Background
The principle of the automatic driving system is that the self information of the vehicle and the surrounding driving environment information are obtained through a sensing system and are transmitted back to a processor for analysis, calculation and processing, and therefore a decision is made to control an execution system to realize acceleration and deceleration of the vehicle and turn light action.
At present, a sensing system in an automatic driving system mainly collects information of a driving environment around a vehicle through hardware such as a vehicle-mounted laser radar and a vehicle-mounted camera and transmits the information back to a vehicle-mounted processor for processing, wherein the sensing system comprises the operations of detecting a target, performing semantic interpretation on environment content, classifying the working condition of the driving environment and the like. In the process, the point cloud returned by the vehicle-mounted laser radar or the picture returned by the vehicle-mounted camera needs to be processed so as to finish the reading of the driving environment information around the vehicle by the sensing system, assist the execution system to react and ensure the safe driving of the vehicle.
In order to reduce the production cost of vehicles with an automatic driving system, researchers are constantly searching whether the driving environment around the vehicle can be analyzed and interpreted through video information collected by a vehicle-mounted camera so as to meet the requirements of a perception system. In general, in-vehicle cameras, i.e., frame rate-based cameras, the returned video information is essentially a pile of a group of pictures, i.e., the processor still analyzes and interprets the pictures. Due to the fact that frame rates of the vehicle-mounted cameras are different and resolutions of returned pictures are different, the calculation amount of the processor is greatly different. Although the high resolution picture may contain more driving environment information, it is necessary to have a high computation cost, which may cause excessive heat generation of the processor chip and further damage to the vehicle components. In addition, when the ordinary vehicle-mounted camera returns pictures, the time interval between frames is in the millisecond order, so that the automobile cannot respond to sudden working conditions in time, and further serious safety problems are caused. If the response time is shortened to increase the frame rate of the general camera, the amount of computation is also increased, which causes excessive heat generation of the chip of the processor.
Disclosure of Invention
The embodiment of the application provides a driving environment sensing method and device, electronic equipment and a storage medium, which can make up for dynamic information missed in the process of returning pictures by a camera according to fixed time, and can also reduce repeated extraction and analysis of a static scene by the camera so as to reduce the amount of calculation of the returned pictures and further reduce the terminal.
The embodiment of the application provides a driving environment sensing method, which comprises the following steps:
acquiring a dynamic information set returned by a dynamic sensing camera within a preset time period; the dynamic perception camera is used for acquiring dynamic information of a dynamic object with changed pixel brightness;
according to the dynamic information set returned within the preset time period, determining a target picture set from the returned picture set returned within the preset time period by the camera;
determining a prediction picture set corresponding to a preset time period according to the dynamic information set and the target picture set;
and determining the driving environment information based on the returned picture set and the predicted picture set.
Furthermore, the preset time period comprises a plurality of sub-time periods, the dynamic information set comprises a plurality of dynamic information sub-sets, the target picture set comprises a plurality of target picture sub-sets, and the sub-time periods, the dynamic information sub-sets and the target picture sub-sets are in one-to-one correspondence;
determining a prediction picture set corresponding to a preset time period according to the dynamic information set and the target picture set, wherein the determining comprises the following steps:
acquiring historical track information corresponding to each sub-time period;
determining a prediction picture subset corresponding to the dynamic information subset according to the historical track information, the dynamic information subset and the target picture subset;
and determining a prediction picture set corresponding to a preset time period according to the prediction picture subset corresponding to the dynamic information subset.
Further, determining a prediction picture set corresponding to a preset time period according to the dynamic information set and the target picture set, including:
and taking the initial track information, the dynamic information set and the target picture set as the input of the picture prediction model, and outputting a prediction picture set and a prediction track information set corresponding to a preset time period.
Further, determining a prediction picture set corresponding to a preset time period according to the dynamic information set and the target picture set, including:
and taking the initial track information, the dynamic information set and the target picture set as the input of the picture prediction model, and outputting a prediction picture set corresponding to a preset time period.
Further, after determining a target picture set from the returned picture sets returned by the camera within a preset time period, the method further includes:
determining a predicted track information set corresponding to the dynamic information set according to the initial track information, the dynamic information set and the target picture set;
determining driving environment information based on the returned picture set and the predicted picture set, including:
and determining the driving environment information based on the predicted track information set, the returned picture set and the predicted picture set.
Further, after obtaining the historical track information corresponding to each sub-period, the method further includes:
determining predicted track information corresponding to each dynamic information subset according to the historical track information, the dynamic information subsets and the target picture subsets;
and determining a predicted track information set corresponding to the dynamic information set according to the predicted track information corresponding to each dynamic information subset.
Further, determining a target picture set from the returned picture sets returned by the camera within a preset time period includes:
if the number of the dynamic information in the dynamic information subset is larger than a preset dynamic information number threshold, determining a picture of the camera at the termination time of the sub-time period from the returned picture set;
if the number of the dynamic information in the dynamic information subset is less than or equal to a preset dynamic information number threshold, determining a picture of the camera at the initial moment of the sub-time period from the returned picture set;
and determining a target picture set according to the picture of the camera at the termination time and/or the picture at the initial time corresponding to each sub-time period.
Correspondingly, the embodiment of the application also provides a driving environment sensing device, which comprises:
the acquisition module is used for acquiring a dynamic information set returned by the dynamic perception camera within a preset time period; the dynamic perception camera is used for acquiring dynamic information of a dynamic object with changed pixel brightness;
the first determining module is used for determining a target picture set from a returned picture set returned by the camera within a preset time period according to the dynamic information set returned within the preset time period;
the second determining module is used for determining a prediction picture set corresponding to a preset time period according to the dynamic information set and the target picture set;
and the third determining module is used for determining the driving environment information based on the returned picture set and the predicted picture set.
Furthermore, the preset time period comprises a plurality of sub-time periods, the dynamic information set comprises a plurality of dynamic information sub-sets, the target picture set comprises a plurality of target picture sub-sets, and the sub-time periods, the dynamic information sub-sets and the target picture sub-sets are in one-to-one correspondence;
the second determining module is used for acquiring historical track information corresponding to each sub-time period; determining a prediction picture subset corresponding to the dynamic information subset according to the historical track information, the dynamic information subset and the target picture subset; and determining a prediction picture set corresponding to a preset time period according to the prediction picture subset corresponding to the dynamic information subset.
Further, the second determining module is configured to take the initial trajectory information, the dynamic information set, and the target picture set as inputs of the picture prediction model, and output a prediction picture set and a prediction trajectory information set corresponding to a preset time period.
Further, the second determining module is configured to take the initial trajectory information, the dynamic information set, and the target picture set as inputs of a picture prediction model, and output a prediction picture set corresponding to a preset time period.
Further, the apparatus further comprises:
the fourth determining module is used for determining a predicted track information set corresponding to the dynamic information set according to the initial track information, the dynamic information set and the target picture set;
and the third determination module is used for determining the driving environment information based on the predicted track information set, the returned picture set and the predicted picture set.
Further, the apparatus further comprises:
the fifth determining module is used for determining the predicted track information corresponding to each dynamic information subset according to the historical track information, the dynamic information subsets and the target picture subsets; and determining a predicted track information set corresponding to the dynamic information set according to the predicted track information corresponding to each dynamic information subset.
Further, the first determining module is configured to determine, from the returned picture set, a picture of the camera at the termination time of the sub-period if the number of pieces of dynamic information in the dynamic information subset is greater than a preset dynamic information number threshold; if the number of the dynamic information in the dynamic information subset is less than or equal to a preset dynamic information number threshold, determining a picture of the camera at the initial moment of the sub-time period from the returned picture set; and determining a target picture set according to the picture of the camera at the termination time and/or the picture at the initial time corresponding to each sub-time period.
Accordingly, an embodiment of the present application further provides an electronic device, which includes a processor and a memory, where the memory stores at least one instruction, at least one program, a code set, or a set of instructions, and the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by the processor to implement the driving environment sensing method described above.
Accordingly, the present application further provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set, or a set of instructions is stored, and the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by a processor to implement the driving environment sensing method.
The embodiment of the application has the following beneficial effects:
the driving environment sensing method, the driving environment sensing device, the electronic device and the storage medium are disclosed in the embodiments of the present application, wherein the method includes acquiring a dynamic information set returned by a dynamic sensing camera within a preset time period, the dynamic sensing camera is used for acquiring dynamic information of a dynamic object with a changed pixel brightness, further determining a target picture set from a returned picture set returned by a camera within the preset time period according to the dynamic information set returned within the preset time period, determining a predicted picture set corresponding to the preset time period according to the dynamic information set and the target picture set, and then determining driving environment information based on the returned picture set and the predicted picture set. Based on the embodiment of the application, the target picture set is determined from the returned picture set returned by the camera within the preset time period according to the returned dynamic information set within the preset time period, so that not only can the dynamic information missed in the process of returning the picture by the camera within the fixed time be made up, but also the repeated extraction and analysis of the static scene by the camera can be reduced, the returned picture can be reduced, and the operation amount of the terminal can be further reduced. In addition, a video frame with a frame rate higher than that returned by the camera can be reconstructed based on the prediction picture set and the returned picture set returned by the camera, and the response time of the automatic driving perception system can be effectively shortened.
Drawings
In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic diagram of an application environment provided by an embodiment of the present application;
FIG. 2 is a flow chart illustrating a driving environment sensing method according to an embodiment of the present disclosure;
fig. 3 is a flowchart illustrating a method for determining a prediction picture set according to an embodiment of the present application;
fig. 4 is a flowchart illustrating a method for determining a prediction picture set according to an embodiment of the present application;
fig. 5 is a schematic flowchart of determining a prediction picture set based on a recurrent neural network according to an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of a driving environment sensing device according to an embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings. It should be apparent that the described embodiment is only one embodiment of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
An "embodiment" as referred to herein relates to a particular feature, structure, or characteristic that may be included in at least one implementation of the present application. In the description of the embodiments of the present application, it should be understood that the terms "upper", "lower", and the like refer to orientations or positional relationships based on those shown in the drawings, and are used for convenience in describing the present application and simplifying the description, but do not indicate or imply that the referred device/system or element must have a specific orientation, be constructed in a specific orientation, and be operated, and thus should not be construed as limiting the present application. The terms "first", "second", "third", "fourth" and "fifth" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first," "second," "third," "fourth," and "fifth" may explicitly or implicitly include one or more of the features. Moreover, the terms "first," "second," "third," "fourth," and "fifth," etc. are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in other sequences than described or illustrated herein. Furthermore, the terms "comprising," "having," and "being," as well as any variations thereof, are intended to cover non-exclusive inclusions.
Dynamic perception cameras, also known as event cameras or dynamic vision sensors, are vision sensors created inspired by the human eye and animal vision system that capture events of dynamic changes in an object. In daily life, the human eyes pay far less attention to the static background than to the dynamic objects in the background, and the dynamic objects can convey more scene information than the static objects. Inspired by the characteristics of human eyes, the dynamic perception camera is designed to capture dynamic information of dynamic objects in a scene, and repeated analysis of static objects is reduced. Different from a common video camera which returns pictures by taking a frame as a unit, the data returned by the dynamic sensing camera mainly comprises three components by taking an event as a unit: the timestamp t, the coordinates (x, y) of the pixel point with the changed brightness, and the polarity (+1, -1) of the brightness change of the pixel point, that is, the dynamic information, which is an event returned by the dynamic perception camera, can be represented as:
event={t,(x,y),sign(dI(x,y)/dt)}
compared with the common video camera which returns pictures in a frame unit, the dynamic sensing camera returns an event, namely dynamic information only when the scene changes, and the speed of the returned event, namely the dynamic information is in the microsecond order and is far higher than that of the millisecond order of the common video camera. Therefore, the dynamic sensing camera can ensure the return speed, and can reduce the repeated analysis of the static object, thereby reducing the computation. However, each frame of picture returned by the conventional video camera contains complete picture information, and the event returned by the motion-sensing camera, i.e. the dynamic information, only contains the dynamic information which changes, i.e. the returned event, i.e. the dynamic information, is too simple. Therefore, the driving environment is sensed by using the dynamic sensing camera singly, and various requirements of the automatic driving sensing system cannot be met.
Based on this, the embodiment of the application provides a driving environment sensing method, a driving environment sensing device, an electronic device and a storage medium, which can solve the problems that a common video camera has a large calculation amount and event information returned by a dynamic sensing camera is too simple.
Please refer to fig. 1, which is a schematic diagram of an application environment according to an embodiment of the present application, including: the perception system 101, the dynamic perception camera 103 and the video camera 105, and the dynamic perception camera 103 and the video camera 105 can be connected with the perception system 101, and in an alternative embodiment, the perception system 101 can be the perception system 101. The sensing system 101 may obtain a dynamic information set, i.e., an event, returned by the dynamic sensing camera 103 within a preset time period, determine a target picture set, i.e., a picture frame, from a returned picture set returned by the video camera 105 within the preset time period according to the dynamic information set returned within the preset time period, determine a predicted picture set corresponding to the preset time period according to the dynamic information set and the target picture set, and then determine driving environment information, i.e., predicted track information and reconstructed video frames, based on the returned picture set and the predicted picture set.
By determining the target picture set from the returned picture set returned by the camera within the preset time period according to the dynamic information set returned within the preset time period, not only can the dynamic information omitted in the process of returning the picture by the camera within the fixed time be made up, but also the repeated extraction and analysis of the static scene by the camera can be reduced, so that the returned picture is reduced, and further, the operation amount of the terminal is reduced. In addition, a video frame with a frame rate higher than that returned by the camera can be reconstructed based on the prediction picture set and the returned picture set returned by the camera, and the response time of the automatic driving perception system can be effectively shortened.
The following describes a specific embodiment of a driving environment sensing method according to the present application, and fig. 2 is a schematic flow chart of a driving environment sensing method according to the embodiment of the present application, and the present specification provides the method operation steps as shown in the embodiment or the flow chart, but more or less operation steps may be included based on conventional or non-inventive labor. The order of steps recited in the embodiments is only one of many possible orders of execution and does not represent the only order of execution, and in actual execution, the steps may be performed sequentially or in parallel as in the embodiments or methods shown in the figures (e.g., in the context of parallel processors or multi-threaded processing). Specifically, as shown in fig. 2, the method may include:
s201: acquiring a dynamic information set returned by a dynamic sensing camera within a preset time period; the dynamic perception camera is used for acquiring dynamic information of a dynamic object with the pixel brightness changing.
In the embodiment of the application, the terminal can control the dynamic sensing camera and the video camera to be both in the open state at the initial moment, and acquire the dynamic information set returned by the dynamic sensing camera within the preset time period. The dynamic information of the dynamic object with the changed pixel brightness returned by the dynamic perception camera within the preset time period is obtained, including the timestamp t, the pixel coordinates (x, y) with the changed brightness, and the polarity (+1, -1) of the brightness change of the pixel, that is, the dynamic information returned by the dynamic perception camera, that is, an event, can be represented as:
event={t,(x,y),sign(dI(x,y)/dt)}
in this embodiment of the application, the preset time period may include a plurality of sub-time periods, and the dynamic information set may include a plurality of dynamic information sub-sets.
In an alternative embodiment, it is assumed that the preset time period is [ t ]0,t1]The frame rate of the camera is 1/delta, and the sub-period comprises [ t [ ]0,t0+Δ]、[t0+Δ,t0+2Δ]...[t0+nΔ,t1]. The terminal can acquire a dynamic information set returned by the dynamic sensing camera within a preset time period and a returned picture set returned by the camera within the preset time period according to the frame rate of the camera.
In particular, the dynamic perception camera may be acquired first for a sub-time period t0,t0+Δ]The dynamic information subset which is internally returned is further acquired when the dynamic perception camera is in [ t ]0+Δ,t0+2Δ]The dynamic information subset returned in the sub-time period is obtained, so that the terminal can obtain the dynamic perception camera at the preset time t0,t1]And (4) internally returning the dynamic information set.
In the embodiment of the application, the dynamic perception camera is acquired at the terminal at t0,t0+Δ]After the dynamic information subset returned in the sub-period, the statistics of the dynamic perception camera at t can be carried out0,t0+Δ]The number of pixels whose luminance changes in the sub-period, that is, the number of pieces of dynamic information, is denoted by N1. Likewise, the dynamic perception camera is acquired at the terminal at [ t ]0+Δ,t0+2Δ]After the dynamic information subset returned in the sub-period, the statistics of the dynamic perception camera at t can be carried out0+Δ,t0+2Δ]The number of pixels whose luminance changes in the sub-period, i.e., the number of dynamic information, is denoted as N2. Thus, the terminal can count each sub-period [ t ]0+(i-1)Δ,t0+iΔ]The number of pixels whose internal brightness changes, that is, the number of motion information is denoted by Ni.
S203: and determining a target picture set from the returned picture sets returned by the camera within the preset time period according to the returned dynamic information set within the preset time period.
In the embodiment of the application, the terminal can determine the target picture set from the picture sets returned by the video camera within the preset time period according to the dynamic information set returned by the dynamic sensing camera within the preset time period. That is, the target picture set may be determined from the picture sets returned by the video camera within the preset time period according to the number of pixels of which the brightness changes in each sub-time period, that is, the number of dynamic information.
In this embodiment of the present application, the target picture set may include target picture sub-sets, and each target picture sub-set corresponds to a sub-time period and a dynamic information sub-set one to one.
In an optional implementation manner, if the number of the dynamic information in the dynamic information subset is greater than a preset dynamic information number threshold, a picture of the camera at the termination time of the sub-time period may be determined from the returned picture set, and then the picture at the termination time corresponding to the sub-time period is taken as a target picture. Namely, when the amount of the dynamic information returned by the dynamic perception camera is large, the terminal acquires the returned picture of the current frame returned by the camera. And if the number of the dynamic information in the dynamic information subset is less than or equal to a preset dynamic information number threshold, determining the picture of the camera at the initial time of the sub-time period from the returned picture set, and further taking the picture at the termination time corresponding to the sub-time period as a target picture set. That is, when the dynamic sensing camera does not return the dynamic information or the amount of the returned dynamic information is small, the terminal does not acquire the returned picture of the camera any more, but adopts the last returned picture.
The following illustrates that a target picture set is determined from picture sets returned by the video camera within a preset time period according to the amount of dynamic information of the dynamic sensing camera within each sub-time period. Assuming that the threshold of the amount of dynamic information is N, if the dynamic sensing camera is at [ t ]0,t0+Δ]The number of dynamic information returned in the sub-period N1, and N1>N, can be represented by0,t0+Δ]The end of the sub-period, i.e. t0+ Δ time camera pass backIs taken as [ t ]0,t0+Δ]The sub-period corresponds to the target picture subset F0. If the dynamic perception camera is at t0+Δ，t0+2Δ]The number of dynamic information returned in the sub-period is N2, and N2 is less than or equal to N, which can be expressed as [ t [ ]0+Δ，t0+2Δ]The initial moment of the sub-period, i.e. t0The picture returned by the camera at the moment of + delta is taken as [ t0+Δ，t0+2Δ]And the target picture subset F1 corresponding to the sub-period, so as to obtain the target picture set.
When the amount of the dynamic information returned by the dynamic sensing camera is greater than a preset dynamic information threshold value, the camera is triggered to return the current frame returned picture, otherwise, a mechanism for obtaining the last frame returned picture returned by the camera is triggered, so that the dynamic information missing in the process that the camera returns the picture according to the fixed time can be compensated, the repeated extraction and analysis of the camera on a static scene can be reduced, the returned picture is reduced, and the operation amount of the terminal is reduced.
S205: and determining a prediction picture set corresponding to a preset time period according to the dynamic information set and the target picture set.
In the embodiment of the application, the terminal can determine the prediction picture set corresponding to the preset time period according to the dynamic information set and the target picture set. Specifically, the prediction picture subset corresponding to each sub-time period may be determined according to the dynamic information subset and the target picture subset corresponding to each sub-time period, and further, the prediction picture set corresponding to the preset time period may be determined according to the prediction picture subset corresponding to each sub-time period.
In an alternative embodiment, the terminal may determine the prediction picture set directly according to the dynamic information set and the target picture set. Fig. 3 is a schematic flowchart of a method for determining a prediction picture set according to an embodiment of the present application, specifically shown in fig. 3:
s301: and acquiring historical track information corresponding to each sub-time period.
S303: and determining a prediction picture subset corresponding to the dynamic information subset according to the historical track information, the dynamic information subset and the target picture subset.
S305: and determining a prediction picture set corresponding to a preset time period according to the prediction picture subset corresponding to the dynamic information subset.
In the embodiment of the application, the terminal can determine the predicted track information set directly according to the dynamic information set and the target picture set. After the historical track information corresponding to each sub-time period is obtained, the terminal can determine the predicted track information corresponding to each dynamic information sub-set according to the historical track information, the dynamic information sub-set and the target picture sub-set, and further determine the predicted track information set corresponding to the dynamic information set according to the predicted track information corresponding to each dynamic information sub-set.
Fig. 4 is a flowchart illustrating a method for determining a prediction picture set according to an embodiment of the present application. In the figure, the terminal may obtain the historical track information corresponding to each sub-period, for example, the sub-period [ t ]0,t0+Δ]Corresponding historical track information, i.e. initial track information sinitSub-period of time [ t ]0+Δ,t0+2Δ]Corresponding historical track information s0.. sub-period t0+nΔ,t1]Corresponding historical track information sn-1And further based on the historical track information si-1Dynamic information subset { e }iAnd target picture subset FiDetermining a subset of predicted pictures { f } corresponding to the subset of dynamic informationiAnd determining a prediction picture set corresponding to a preset time period according to the prediction picture subset corresponding to the dynamic information subset.
In another alternative embodiment, the terminal may determine the prediction picture set and the prediction track information set by using a picture prediction model. For example, the initial trajectory information, the dynamic information set, and the target picture set may be used as inputs of a picture prediction model, and a prediction picture set and a prediction trajectory information set corresponding to a preset time period may be output.
Specifically, the picture prediction model may be embodied as a recurrent neural network, fig. 5 is a schematic flowchart of a process for determining a predicted picture set based on the recurrent neural network according to an embodiment of the present application,
the input and output variables of the recurrent neural network at the ith frame are defined as follows:
inputting:
1.Fi: sub-period of time [ t ]0+(i-1)Δ,t0+iΔ]A corresponding target picture subset;
2.{ei}: sub-period of time [ t ]0+(i-1)Δ,t0+iΔ]A corresponding subset of dynamic information;
3.si-1: sub-period of time [ t ]0+(i-1)Δ,t0+iΔ]Corresponding historical track information; wherein s isinitSub-period of time [ t ]0,t0+Δ]Corresponding historical track information, namely initial track information;
and (3) outputting:
1.si: sub-period of time [ t ]0+(i-1)Δ,t0+iΔ]Corresponding predicted trajectory information; the predicted trajectory information siIs a sub-period of time t0+iΔ,t0+(i+1)Δ]Corresponding historical track information;
2.{fi}: sub-period of time [ t ]0+(i-1)Δ,t0+iΔ]The corresponding prediction picture.
The forward propagation structure of the recurrent neural network is as follows:
wherein, U1Representing an input layer FiWeight to hidden layer, U2Representing an input layer eiWeight to hidden layer, U3Representing an input layer si-1Weight to hidden layer, V1Representing hidden layers to output fiWeight of (V)2Representing hidden layers to output siWeight of (g), gi(i =1,2, 3) are different activation functions, hiThe hidden layers are marked with a circle in fig. 5.
In another alternative embodiment, the terminal may determine only the prediction picture set using the picture prediction model. The initial trajectory information, the dynamic information set and the target picture set can be used as input of a picture prediction model, and a prediction picture set corresponding to a preset time period is output.
Specifically, the picture prediction model may be embodied as a recurrent neural network, and the input and output variables of the recurrent neural network at the ith frame are defined as follows:
inputting:
1.Fi: sub-period of time [ t ]0+(i-1)Δ,t0+iΔ]A corresponding target picture subset, i =1,2.. n;
2.{ei}: sub-period of time [ t ]0+(i-1)Δ,t0+iΔ]A corresponding subset of dynamic information, i =1,2.. n;
3.sinit: initial track information;
and (3) outputting:
1.{fi}: sub-period of time [ t ]0+(i-1)Δ,t0+iΔ]Corresponding prediction picture, i =1,2.. n.
S207: and determining the driving environment information based on the returned picture set and the predicted picture set.
In the embodiment of the application, the terminal can determine the driving environment information based on the returned picture set and the predicted picture set.
In an optional implementation manner, a predicted picture set output by the recurrent neural network and a returned picture set returned by the camera can be utilized to reconstruct a video frame with a frame rate higher than that returned by the camera, so that the response time of the automatic driving sensing system can be effectively shortened.
In this embodiment of the application, after the target picture set is determined from the returned picture set returned by the camera within the preset time period as described above, the terminal may determine a predicted trajectory information set corresponding to the dynamic information set according to the initial trajectory information, the dynamic information set, and the target picture set, and then determine the driving environment information based on the predicted trajectory information set, the returned picture set, and the predicted picture set.
In an optional implementation manner, a predicted picture set output by the recurrent neural network and a returned picture set returned by the camera can be utilized to reconstruct a video frame with a frame rate higher than that returned by the camera, so that the response time of the automatic driving sensing system can be effectively shortened. In addition, the motion trail of the dynamic object in the scene can be predicted by utilizing the predicted trail information set output by the recurrent neural network, so that the driving environment information is enriched, and the accuracy of the driving environment information is improved.
By adopting the driving environment sensing method provided by the embodiment of the application, when the amount of the dynamic information returned by the dynamic sensing camera is larger than the preset dynamic information threshold value, the current frame returned picture returned by the camera is triggered to be acquired, otherwise, the mechanism for acquiring the last frame returned picture returned by the camera is triggered, so that the dynamic information missed in the process that the camera returns the picture according to the fixed time can be compensated, the repeated extraction and analysis of the camera on the static scene can be reduced, the returned picture can be reduced, and the operation amount of the terminal can be reduced. In addition, the predicted picture set output by the recurrent neural network and the returned picture set returned by the camera are utilized to reconstruct a video frame with a higher frame rate compared with the returned frame rate of the camera, so that the response time of the automatic driving perception system can be effectively shortened. In addition, the motion trail of the dynamic object in the scene can be predicted by utilizing the predicted trail information set output by the recurrent neural network, so that the driving environment information is enriched, and the accuracy of the driving environment information is improved.
Fig. 6 is a schematic structural diagram of the driving environment sensing device provided in the embodiment of the present application, and as shown in fig. 6, the driving environment sensing device may include:
the obtaining module 601 may be configured to obtain a dynamic information set returned by the dynamic sensing camera within a preset time period; the dynamic perception camera is used for acquiring dynamic information of a dynamic object with changed pixel brightness;
the first determining module 603 may be configured to determine, according to the dynamic information set returned within the preset time period, a target picture set from the returned picture set returned within the preset time period by the camera;
the second determining module 605 may be configured to determine a prediction picture set corresponding to a preset time period according to the dynamic information set and the target picture set;
the third determination module 607 may be configured to determine the driving environment information based on the returned picture set and the predicted picture set.
In this embodiment of the present application, the preset time period may include a plurality of sub-time periods, the dynamic information set may include a plurality of dynamic information sub-sets, the target picture set may include a plurality of target picture sub-sets, and the sub-time periods, the dynamic information sub-sets and the target picture sub-sets correspond to one another;
the second determining module 605 may be configured to obtain historical track information corresponding to each sub-period; determining a prediction picture subset corresponding to the dynamic information subset according to the historical track information, the dynamic information subset and the target picture subset; and determining a prediction picture set corresponding to a preset time period according to the prediction picture subset corresponding to the dynamic information subset.
The second determining module 605 may be configured to take the initial trajectory information, the dynamic information set, and the target picture set as inputs of a picture prediction model, and output a prediction picture set and a prediction trajectory information set corresponding to a preset time period.
The second determining module 605 may be configured to take the initial trajectory information, the dynamic information set, and the target picture set as inputs of a picture prediction model, and output a prediction picture set corresponding to a preset time period.
In this embodiment, the apparatus may further include:
the fourth determining module 609 is configured to determine, according to the initial trajectory information, the dynamic information set, and the target picture set, a predicted trajectory information set corresponding to the dynamic information set;
the third determination module 607 is configured to determine the driving environment information based on the predicted trajectory information set, the returned picture set, and the predicted picture set.
In this embodiment, the apparatus may further include:
the fifth determining module is used for determining the predicted track information corresponding to each dynamic information subset according to the historical track information, the dynamic information subsets and the target picture subsets; and determining a predicted track information set corresponding to the dynamic information set according to the predicted track information corresponding to each dynamic information subset.
In this embodiment of the application, the first determining module 603 is configured to determine, from the returned picture set, a picture of the camera at the termination time of the sub-period if the number of pieces of dynamic information in the dynamic information subset is greater than a preset dynamic information number threshold; if the number of the dynamic information in the dynamic information subset is less than or equal to a preset dynamic information number threshold, determining a picture of the camera at the initial moment of the sub-time period from the returned picture set; and determining a target picture set according to the picture of the camera at the termination time and/or the picture at the initial time corresponding to each sub-time period.
The device and method embodiments in the embodiments of the present application are based on the same application concept.
By adopting the driving environment sensing device provided by the embodiment of the application, when the amount of the dynamic information returned by the dynamic sensing camera is larger than the preset dynamic information threshold value, the current frame returned picture returned by the camera is triggered to be acquired, otherwise, the mechanism for acquiring the last frame returned picture returned by the camera is triggered, so that not only can the dynamic information missed in the process that the camera returns the picture according to the fixed time be made up, but also the repeated extraction and analysis of the static scene by the camera can be reduced, so that the returned picture is reduced, and further, the operation amount of the terminal is reduced. In addition, the predicted picture set output by the recurrent neural network and the returned picture set returned by the camera are utilized to reconstruct a video frame with a higher frame rate compared with the returned frame rate of the camera, so that the response time of the automatic driving perception system can be effectively shortened. In addition, the motion trail of the dynamic object in the scene can be predicted by utilizing the predicted trail information set output by the recurrent neural network, so that the driving environment information is enriched, and the accuracy of the driving environment information is improved.
The present invention further provides an electronic device, which may be disposed in a server to store at least one instruction, at least one program, a code set, or a set of instructions related to implementing a driving environment sensing method in the method embodiments, where the at least one instruction, the at least one program, the code set, or the set of instructions is loaded from the memory and executed to implement the driving environment sensing method.
The present application further provides a storage medium, which may be disposed in a server to store at least one instruction, at least one program, a code set, or a set of instructions related to implementing a driving environment sensing method in the method embodiment, where the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by the processor to implement the driving environment sensing method.
Optionally, in this embodiment, the storage medium may be located in at least one network server of a plurality of network servers of a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, a storage medium including: various media that can store program codes, such as a usb disk, a Read-only Memory (ROM), a removable hard disk, a magnetic disk, or an optical disk.
As can be seen from the above embodiments of the driving environment sensing method, apparatus, electronic device, or storage medium provided by the present application, the method in the present application includes acquiring a dynamic information set returned by a dynamic sensing camera within a preset time period, where the dynamic sensing camera is used to acquire dynamic information of a dynamic object whose pixel brightness changes, and further determining a target picture set from a returned picture set returned by a camera within the preset time period according to the dynamic information set returned within the preset time period, and determining a predicted picture set corresponding to the preset time period according to the dynamic information set and the target picture set, and then determining driving environment information based on the returned picture set and the predicted picture set. Based on the embodiment of the application, the target picture set is determined from the returned picture set returned by the camera within the preset time period according to the returned dynamic information set within the preset time period, so that not only can the dynamic information missed in the process of returning the picture by the camera within the fixed time be made up, but also the repeated extraction and analysis of the static scene by the camera can be reduced, the returned picture can be reduced, and the operation amount of the terminal can be further reduced. In addition, a video frame with a frame rate higher than that returned by the camera can be reconstructed based on the prediction picture set and the returned picture set returned by the camera, and the response time of the automatic driving perception system can be effectively shortened.
It should be noted that: the foregoing sequence of the embodiments of the present application is for description only and does not represent the superiority and inferiority of the embodiments, and the specific embodiments are described in the specification, and other embodiments are also within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in the order of execution in different embodiments and achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown or connected to enable the desired results to be achieved, and in some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
All the embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment is described with emphasis on differences from other embodiments. Especially, for the embodiment of the device, since it is based on the embodiment similar to the method, the description is simple, and the relevant points can be referred to the partial description of the method embodiment.
While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention.
Claims (12)
1. A driving environment sensing method, comprising:
acquiring a dynamic information set returned by a dynamic sensing camera within a preset time period; the dynamic perception camera is used for acquiring dynamic information of a dynamic object with changed pixel brightness;
according to the dynamic information set returned in the preset time period, determining a target picture set from the returned picture set returned by the camera in the preset time period; the preset time period comprises a plurality of sub-time periods, the dynamic information set comprises a plurality of dynamic information sub-sets, the target picture set comprises a plurality of target picture sub-sets, and the sub-time periods, the dynamic information sub-sets and the target picture sub-sets are in one-to-one correspondence;
determining a prediction picture set corresponding to the preset time period according to the dynamic information set and the target picture set;
reconstructing a video frame and determining driving environment information based on the return picture set and the prediction picture set, wherein the return frame rate corresponding to the video frame is higher than the initial return frame rate of the camera;
determining a prediction picture set corresponding to the preset time period according to the dynamic information set and the target picture set, including:
acquiring historical track information corresponding to each sub-time period;
determining a prediction picture subset corresponding to the dynamic information subset according to the historical track information, the dynamic information subset and the target picture subset;
and determining the prediction picture set corresponding to the preset time period according to the prediction picture subset corresponding to the dynamic information subset.
2. The method according to claim 1, wherein the determining a target picture set from the returned picture sets returned by the camera within the preset time period according to the dynamic information sets returned within the preset time period comprises:
if the number of the dynamic information in the dynamic information subset is larger than a preset dynamic information number threshold, determining a picture of the camera at the termination time of the sub-time period from the returned picture set;
if the number of the dynamic information in the dynamic information subset is less than or equal to a preset dynamic information number threshold, determining a picture of the camera at the initial moment of the sub-time period from the returned picture set;
and determining the target picture set according to the picture of the camera at the termination time and/or the picture at the initial time corresponding to each sub-time period.
3. The method according to claim 1, wherein the determining the prediction picture set corresponding to the preset time period according to the dynamic information set and the target picture set comprises:
and taking the initial track information, the dynamic information set and the target picture set as the input of a picture prediction model, and outputting a prediction picture set and a prediction track information set corresponding to the preset time period.
4. The method according to claim 1, wherein after determining the target set of pictures from the returned set of pictures returned by the camera within the preset time period, further comprising:
determining a predicted track information set corresponding to the dynamic information set according to the initial track information, the dynamic information set and the target picture set;
the determining driving environment information based on the returned picture set and the predicted picture set comprises:
and determining running environment information based on the predicted track information set, the returned picture set and the predicted picture set.
5. The method according to claim 1, wherein after the obtaining of the historical track information corresponding to each sub-period, the method further comprises:
determining predicted track information corresponding to each dynamic information subset according to the historical track information, the dynamic information subsets and the target picture subsets;
and determining a predicted track information set corresponding to the dynamic information set according to the predicted track information corresponding to each dynamic information subset.
6. A running environment sensing apparatus, comprising:
the acquisition module is used for acquiring a dynamic information set returned by the dynamic perception camera within a preset time period; the dynamic perception camera is used for acquiring dynamic information of a dynamic object with changed pixel brightness;
the first determining module is used for determining a target picture set from returned picture sets returned by the camera within the preset time period according to the dynamic information set returned within the preset time period; the preset time period comprises a plurality of sub-time periods, the dynamic information set comprises a plurality of dynamic information sub-sets, the target picture set comprises a plurality of target picture sub-sets, and the sub-time periods, the dynamic information sub-sets and the target picture sub-sets are in one-to-one correspondence;
the second determining module is used for determining a prediction picture set corresponding to the preset time period according to the dynamic information set and the target picture set;
a third determining module, configured to reconstruct a video frame and determine driving environment information based on the returned picture set and the predicted picture set, where a returned frame rate corresponding to the video frame is higher than an initial returned frame rate of the camera;
the second determining module is configured to obtain historical track information corresponding to each sub-time period; determining a prediction picture subset corresponding to the dynamic information subset according to the historical track information, the dynamic information subset and the target picture subset; and determining the prediction picture set corresponding to the preset time period according to the prediction picture subset corresponding to the dynamic information subset.
7. The apparatus of claim 6,
the first determining module is configured to determine, from the returned picture set, a picture of the camera at the termination time of the sub-period if the number of pieces of dynamic information in the dynamic information sub-set is greater than a preset dynamic information number threshold;
if the number of the dynamic information in the dynamic information subset is less than or equal to a preset dynamic information number threshold, determining a picture of the camera at the initial moment of the sub-time period from the returned picture set;
and determining the target picture set according to the picture of the camera at the termination time and/or the picture at the initial time corresponding to each sub-time period.
8. The apparatus of claim 6,
the second determining module is configured to take the initial trajectory information, the dynamic information set, and the target picture set as inputs of a picture prediction model, and output a prediction picture set and a prediction trajectory information set corresponding to the preset time period.
9. The apparatus of claim 6, further comprising:
the fourth determining module is used for determining a predicted track information set corresponding to the dynamic information set according to the initial track information, the dynamic information set and the target picture set;
the third determining module is configured to determine driving environment information based on the predicted trajectory information set, the returned picture set, and the predicted picture set.
10. The apparatus of claim 6, further comprising:
a fifth determining module, configured to determine, according to the historical track information, the dynamic information subset, and the target picture subset, predicted track information corresponding to each dynamic information subset; and determining a predicted track information set corresponding to the dynamic information set according to the predicted track information corresponding to each dynamic information subset.
11. An electronic device, comprising a processor and a memory, wherein at least one instruction, at least one program, a set of codes, or a set of instructions is stored in the memory, and wherein the at least one instruction, the at least one program, the set of codes, or the set of instructions is loaded and executed by the processor to implement the driving environment awareness method of any one of claims 1-5.
12. A computer-readable storage medium, wherein at least one instruction, at least one program, a set of codes, or a set of instructions is stored, which is loaded and executed by a processor to implement a driving environment awareness method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110880924.5A CN113326820B (en) | 2021-08-02 | 2021-08-02 | Driving environment sensing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110880924.5A CN113326820B (en) | 2021-08-02 | 2021-08-02 | Driving environment sensing method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113326820A CN113326820A (en) | 2021-08-31 |
| CN113326820B true CN113326820B (en) | 2021-11-30 |
Family
ID=77426755
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110880924.5A Active CN113326820B (en) | 2021-08-02 | 2021-08-02 | Driving environment sensing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113326820B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111479087A (en) * | 2019-01-23 | 2020-07-31 | 北京奇虎科技有限公司 | 3D monitoring scene control method and device, computer equipment and storage medium |
| CN112950786A (en) * | 2021-03-01 | 2021-06-11 | 哈尔滨理工大学 | Vehicle three-dimensional reconstruction method based on neural network |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102584501B1 (en) * | 2018-10-05 | 2023-10-04 | 삼성전자주식회사 | Method for recognizing object and autonomous driving device therefor |
| CN109583151B (en) * | 2019-02-20 | 2023-07-21 | 阿波罗智能技术(北京)有限公司 | Method and device for predicting running track of vehicle |
| CN112068603A (en) * | 2020-08-10 | 2020-12-11 | 上海交通大学 | Unmanned vehicle following system and method based on event camera |
| CN113096158A (en) * | 2021-05-08 | 2021-07-09 | 北京灵汐科技有限公司 | Moving object identification method and device, electronic equipment and readable storage medium |
-
2021
- 2021-08-02 CN CN202110880924.5A patent/CN113326820B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111479087A (en) * | 2019-01-23 | 2020-07-31 | 北京奇虎科技有限公司 | 3D monitoring scene control method and device, computer equipment and storage medium |
| CN112950786A (en) * | 2021-03-01 | 2021-06-11 | 哈尔滨理工大学 | Vehicle three-dimensional reconstruction method based on neural network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113326820A (en) | 2021-08-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20240112051A1 (en) | Machine learning models operating at different frequencies for autonomous vehicles | |
| US7555046B2 (en) | Method and system for searching and verifying magnitude change events in video surveillance | |
| CN113286194A (en) | Video processing method and device, electronic equipment and readable storage medium | |
| CN104205169A (en) | Method of estimating optical flow on the basis of an asynchronous light sensor | |
| US11501536B2 (en) | Image processing method, an image processing apparatus, and a surveillance system | |
| CN111523347A (en) | Image detection method and device, computer equipment and storage medium | |
| US20210073997A1 (en) | Future semantic segmentation prediction using 3d structure | |
| US20210035325A1 (en) | Pose estimation method, pose estimation apparatus, and training method for pose estimation | |
| CN111768438B (en) | Image processing method, device, equipment and computer readable storage medium | |
| CN111209811B (en) | Method and system for detecting eyeball attention position in real time | |
| CN110572636B (en) | Camera contamination detection method and device, storage medium and electronic equipment | |
| CN113177432A (en) | Head pose estimation method, system, device and medium based on multi-scale lightweight network | |
| CN113326820B (en) | Driving environment sensing method and device, electronic equipment and storage medium | |
| CN112685125B (en) | Theme switching method and device of application program and computer readable storage medium | |
| CN114051116A (en) | Video monitoring method, device and system for driving test vehicle | |
| CN115908618A (en) | Reconstructed image generation method, device, equipment and medium based on pulse data | |
| US11749004B2 (en) | Event detector and method of generating textural image based on event count decay factor and net polarity | |
| CN115871679A (en) | Driver fatigue detection method, driver fatigue detection device, electronic device, and medium | |
| JP2024516642A (en) | Behavior detection method, electronic device and computer-readable storage medium | |
| CN111784736A (en) | Moving small target real-time detection system and method based on single-pixel parallel computation | |
| US11810307B2 (en) | Facilitating object tracking for portable camera images | |
| EP4344227A1 (en) | Video frame interpolation method and apparatus, and device | |
| US20230368520A1 (en) | Fast object detection in video via scale separation | |
| Liu et al. | Dynamic Multi-ROI Parallel Inference Architecture for Online Video | |
| CN117372469A (en) | Target tracking system, method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |