CN117831156A - Data playback method and system, server and medium - Google Patents

Abstract

The application provides a data playback method and system, a server and a medium, wherein the method comprises the steps of responding to a data playback request, acquiring time sequence control information and intelligent equipment input data corresponding to the time sequence control information; based on the input data of the intelligent equipment, executing server data processing operation to obtain server output data; adding a time stamp of the server output data to the server output data based on the time stamp of the intelligent device output data to obtain server to-be-played-back data consistent with the time stamp of the intelligent device output data; based on the timestamp of the output data of the server, the data to be played back of the server is played back, the fact that the server plays back according to the time sequence of each data operation executed by the intelligent device is achieved, the fact that the output data of the server is consistent with the output data of the intelligent device is guaranteed, and accordingly the automatic driving system can be adjusted accurately.

Description

Data playback method and system, server and medium

Technical Field

The application relates to the technical field of data playback, and particularly provides a data playback method and system, a server and a medium.

Background

In the development and practical application process of the autopilot system, some data collected by an intelligent device installed with the autopilot system are generally utilized, so that after corresponding operations are performed based on the collected data to obtain output data, corresponding adjustment strategies and the like are made for the autopilot system based on the output data.

However, when the server plays back some data collected by the smart device, the result output by the server is inconsistent with the output result of the smart device often because of different hardware, different operating systems, different machine loads and the like between the smart device and the server.

Disclosure of Invention

In order to overcome the above-mentioned drawbacks, the present application is provided to provide a data playback method and system, a server and a medium that solve or at least partially solve the technical problem that the result output by the server is inconsistent with the smart device.

In a first aspect, the present application provides a data playback method, applied to a server, including:

responding to a data playback request, and acquiring time sequence control information; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment;

Acquiring intelligent equipment input data corresponding to the time sequence control information based on the time sequence control information; the intelligent equipment input data is obtained based on sensing data acquired by a sensor of the intelligent equipment; the perception data comprises at least one of image data, laser radar data or millimeter wave radar data;

based on the intelligent equipment input data, executing server data processing operation to obtain server output data;

adding a time stamp of the server output data to the server output data based on the time stamp of the intelligent device output data to obtain server to-be-played-back data consistent with the time stamp of the intelligent device output data;

and playing back the data to be played back of the server based on the timestamp of the output data of the server.

In a second aspect, the present application provides a data playback method applied to a smart device, the smart device including a sensor, the data playback method including:

responding to a data processing request, and obtaining intelligent equipment input data based on sensing data acquired by a sensor of the intelligent equipment, wherein the sensing data comprises at least one of image data, laser radar data or millimeter wave radar data;

Performing data processing on the intelligent device input data to obtain intelligent device output data, and timing control information corresponding to the intelligent device output data and the intelligent device input data; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment;

and sending the intelligent device input data, the intelligent device output data and the time sequence control information to a server, so that when the server responds to a data playback request, obtaining the server to-be-played-back data consistent with the time stamp of the intelligent device output data based on the acquired time sequence control information, and playing back the server to-be-played-back data based on the time stamp of the server output data.

In a third aspect, a data playback method is provided, applied to a data playback system including an intelligent device and a server, where information interaction is performed between the intelligent device and the server, the data playback method includes:

the intelligent equipment responds to the data processing request, and obtains intelligent equipment input data based on sensing data acquired by a sensor of the intelligent equipment; performing data processing on the intelligent device input data to obtain intelligent device output data, and timing control information corresponding to the intelligent device output data and the intelligent device input data; transmitting the intelligent device input data, the intelligent device output data and the time sequence control information to a server; wherein the perception data comprises at least one of image data, laser radar data or millimeter wave radar data; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment;

The server responds to the data playback request and acquires time sequence control information; acquiring intelligent equipment input data corresponding to the time sequence control information based on the time sequence control information; acquiring intelligent equipment input data corresponding to the time sequence control information based on the time sequence control information; adding a time stamp of the server output data to the server output data based on the time stamp of the intelligent device output data to obtain server to-be-played-back data consistent with the time stamp of the intelligent device output data; based on the timestamp of the output data of the server, playing back the data to be played back of the server; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment; the intelligent equipment input data is obtained based on sensing data acquired by a sensor of the intelligent equipment; the perception data includes at least one of image data, laser radar data, or millimeter wave radar data.

In a fourth aspect, there is provided a server comprising a processor and a memory, the memory being adapted to store a plurality of program codes, the program codes being adapted to be loaded and executed by the processor to perform the data playback method of any one of the preceding claims.

In a fifth aspect, there is provided a smart device comprising a processor and a memory, the memory being adapted to store a plurality of program code, the program code being adapted to be loaded and executed by the processor to perform the data playback method of any one of the preceding claims.

In a sixth aspect, a data playback system is provided, the data playback system comprising a smart device and a server;

the intelligent device is configured to execute the data playback method applied to the intelligent device;

the server is configured to perform the above-described data playback method applied to the server.

In a seventh aspect, there is provided a computer readable storage medium storing a plurality of program codes adapted to be loaded and executed by a processor to perform the data playback method of any one of the above.

1. A data playback method, characterized by being applied to a server, comprising:

responding to a data playback request, and acquiring time sequence control information; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment;

acquiring intelligent equipment input data corresponding to the time sequence control information based on the time sequence control information; the intelligent equipment input data is obtained based on sensing data acquired by a sensor of the intelligent equipment; the perception data comprises at least one of image data, laser radar data or millimeter wave radar data;

Based on the intelligent equipment input data, executing server data processing operation to obtain server output data;

adding a time stamp of the server output data to the server output data based on the time stamp of the intelligent device output data to obtain server to-be-played-back data consistent with the time stamp of the intelligent device output data;

and playing back the data to be played back of the server based on the timestamp of the output data of the server.

The method for playing back data according to the scheme 1, wherein the timing control information further includes intelligent device process information;

the method further comprises the steps of:

if there are a plurality of timing control information to be acquired, the acquiring the timing control information includes:

determining the acquisition sequence of the time sequence control information to be acquired based on the intelligent equipment process information corresponding to each of the time sequence control information to be acquired;

and sequentially acquiring one of the plurality of timing control information to be acquired as the timing control information based on the acquisition order.

The method for playing back data according to the scheme 3, is characterized in that determining the acquisition order of the plurality of timing control information based on the intelligent device process information corresponding to each of the plurality of timing control information to be acquired, comprising:

If the plurality of time sequence control information to be acquired are all acquired based on the same type of equipment data processing operation, determining the acquisition sequence according to the execution sequence in the intelligent equipment process information and the sequence from small to large;

the device data processing operation comprises the operation of performing data processing on sensing data from the same type of sensor to obtain sensing results of the type of sensor, or the operation of fusing sensing results of different types of sensors to obtain fused sensing results.

The method for playing back data according to the scheme 4 is characterized in that determining the acquisition order of the plurality of timing control information based on the intelligent device process information corresponding to each of the plurality of timing control information to be acquired, including:

if the plurality of time sequence control information to be acquired are respectively acquired based on different types of equipment data processing operations, determining the acquisition sequence according to the starting time in the intelligent equipment process information and the sequence from front to back of the time;

the device data processing operation comprises the operation of performing data processing on sensing data from the same type of sensor to obtain sensing results of the type of sensor, or the operation of fusing sensing results of different types of sensors to obtain fused sensing results.

The method for playing back data according to the scheme 5, wherein determining the acquisition order of the plurality of timing control information based on the smart device process information corresponding to each of the plurality of timing control information to be acquired includes:

if a part of the time sequence control information to be acquired in the plurality of time sequence control information to be acquired is acquired based on the same type of equipment data processing operation, and the rest of the time sequence control information to be acquired is acquired based on different types of equipment data processing operation respectively, determining the acquisition order according to the starting time and the execution order in the intelligent equipment process information;

the device data processing operation comprises the operation of performing data processing on sensing data from the same type of sensor to obtain sensing results of the type of sensor, or the operation of fusing sensing results of different types of sensors to obtain fused sensing results.

The data playback method according to claim 1, further comprising:

and storing the data to be played back of the server to the data information base.

The method for playing back data according to the scheme 7 is characterized in that the timing control information further comprises inter-frame ordering information, the inter-frame ordering information comprises first inter-frame ordering information of a plurality of sensing data when the sensing data from the same type of sensor are processed to obtain the sensing result of the type, and/or second inter-frame ordering information of a plurality of sensing results when the sensing results of different types of sensors are fused to obtain the fused sensing result;

Based on the timing control information, acquiring the intelligent device input data includes:

and sequentially acquiring a plurality of intelligent device input data corresponding to the time stamp and the inter-frame ordering information according to the time stamp of the intelligent device input data and the inter-frame ordering information.

Scheme 8. A data playback method, characterized in that it is applied to a smart device, said smart device comprising a sensor, said method comprising:

responding to a data processing request, and obtaining intelligent equipment input data based on sensing data acquired by a sensor of the intelligent equipment, wherein the sensing data comprises at least one of image data, laser radar data or millimeter wave radar data;

performing data processing on the intelligent device input data to obtain intelligent device output data, and timing control information corresponding to the intelligent device output data and the intelligent device input data; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment;

and sending the intelligent device input data, the intelligent device output data and the time sequence control information to a server, so that when the server responds to a data playback request, obtaining the server to-be-played-back data consistent with the time stamp of the intelligent device output data based on the acquired time sequence control information, and playing back the server to-be-played-back data based on the time stamp of the server output data.

The scheme 9 is characterized by being applied to a data playback system comprising intelligent equipment and a server, wherein information interaction is carried out between the intelligent equipment and the server, and the method comprises the following steps:

the intelligent equipment responds to the data processing request, and obtains intelligent equipment input data based on sensing data acquired by a sensor of the intelligent equipment; performing data processing on the intelligent device input data to obtain intelligent device output data, and timing control information corresponding to the intelligent device output data and the intelligent device input data; transmitting the intelligent device input data, the intelligent device output data and the time sequence control information to a server; wherein the perception data comprises at least one of image data, laser radar data or millimeter wave radar data; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment;

the server responds to the data playback request and acquires time sequence control information; acquiring intelligent equipment input data corresponding to the time sequence control information based on the time sequence control information; acquiring intelligent equipment input data corresponding to the time sequence control information based on the time sequence control information; adding a time stamp of the server output data to the server output data based on the time stamp of the intelligent device output data to obtain server to-be-played-back data consistent with the time stamp of the intelligent device output data; based on the timestamp of the output data of the server, playing back the data to be played back of the server; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment; the intelligent equipment input data is obtained based on sensing data acquired by a sensor of the intelligent equipment; the perception data includes at least one of image data, laser radar data, or millimeter wave radar data.

A server comprising a processor and a memory, the memory being adapted to store a plurality of program codes, the program codes being adapted to be loaded and executed by the processor to perform the data playback method of any one of schemes 1 to 7.

Scheme 11. A smart device characterized in that it comprises a processor and a memory, said memory being adapted to store a plurality of program codes, said program codes being adapted to be loaded and executed by said processor to perform the data playback method of scheme 8.

Scheme 12. A data playback system, characterized by comprising an intelligent device and a server;

the smart device configured to perform the data playback method of aspect 8;

the server is configured to perform the data playback method of any one of aspects 1 to 7.

A computer readable storage medium, characterized in that a plurality of program codes are stored, said program codes being adapted to be loaded and executed by a processor to perform the data playback method of any one of the schemes 1 to 9.

The technical scheme has at least one or more of the following beneficial effects:

in the technical scheme of implementing the application, a server responds to a data playback request to acquire time sequence control information, and based on the time sequence control information, intelligent equipment input data corresponding to the time sequence control information is acquired; based on the intelligent equipment input data, executing server data processing operation to obtain server output data; adding a time stamp of the server output data to the server output data based on the time stamp of the intelligent device output data to obtain server to-be-played-back data consistent with the time stamp of the intelligent device output data; based on the timestamp of the output data of the server, the data to be played back of the server is played back, so that the server plays back according to the time sequence of each data operation executed by the intelligent device, the output data of the server is consistent with the output data of the intelligent device, and the corresponding adjustment strategy is conveniently and accurately made for the automatic driving system.

Drawings

The disclosure of the present application will become more readily understood with reference to the accompanying drawings. As will be readily appreciated by those skilled in the art: these drawings are for illustrative purposes only and are not intended to limit the scope of the present application. Moreover, like numerals in the figures are used to designate like parts, wherein:

FIG. 1 is a flow chart of the main steps of a data playback method according to one embodiment of the present application;

FIG. 2 is a schematic structural view of a perception system in an autonomous vehicle;

FIG. 3 is a data flow dependency graph;

FIG. 4 is a flow diagram of a particular implementation of a data playback method according to one embodiment of the present application;

fig. 5 is a main structural block diagram of a data playback apparatus according to an embodiment of the present application;

FIG. 6 is a block diagram of the main structure of a server according to one embodiment of the present application;

FIG. 7 is a block diagram of the main structure of a server according to one embodiment of the present application;

fig. 8 is a main structural block diagram of a data playback system according to an embodiment of the present application.

Detailed Description

Some embodiments of the present application are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are merely for explaining the technical principles of the present application, and are not intended to limit the scope of the present application.

In the description of the present application, a "module," "processor" may include hardware, software, or a combination of both. A module may comprise hardware circuitry, various suitable sensors, communication ports, memory, or software components, such as program code, or a combination of software and hardware. The processor may be a central processor, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and/or signal processing functions. The processor may be implemented in software, hardware, or a combination of both. Non-transitory computer readable storage media include any suitable medium that can store program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, and the like. The term "a and/or B" means all possible combinations of a and B, such as a alone, B alone or a and B. The term "at least one A or B" or "at least one of A and B" has a meaning similar to "A and/or B" and may include A alone, B alone or A and B. The singular forms "a", "an" and "the" include plural referents.

An autopilot system (Automated Driving Systems, ADS), which means that the system will continue to perform all dynamic driving tasks (Dynamic Driving Task, DDT) within its design operating range (Operational Domain Design, ODD). That is, the machine system is allowed to fully take over the task of the vehicle autonomous handling under the prescribed appropriate driving scenario conditions, i.e., the system is activated when the vehicle satisfies the ODD condition, which replaces the human driver as the driving subject of the vehicle. Among them, the dynamic driving task DDT refers to continuous lateral (left and right steering) and longitudinal motion control (acceleration, deceleration, uniform) of the vehicle and detection and response of targets and events in the running environment of the vehicle. The design operation range ODD refers to a condition under which the automatic driving system can safely operate, and the set condition may include a geographical location, a road type, a speed range, weather, time, country and local traffic laws and regulations, and the like.

In the development and practical application process of the autopilot system, some data collected by an intelligent device installed with the autopilot system are generally utilized, so that after corresponding operations are performed based on the collected data to obtain output data, corresponding adjustment strategies and the like are made for the autopilot system based on the output data. The intelligent device can specifically comprise a driving device, an automatic driving vehicle, an intelligent vehicle, a robot, an unmanned plane and the like. In some embodiments, the smart device further comprises at least one sensor for acquiring sensory data. The sensor is communicatively coupled to any of the types of processors referred to herein.

However, when the server plays back some data collected by the smart device, the result output by the server is inconsistent with the output result of the smart device often because of different hardware, different operating systems, different machine loads and the like between the smart device and the server.

Therefore, in order to solve the above technical problems, the present application provides the following technical solutions:

referring to fig. 1, fig. 1 is a schematic flow chart of main steps of a data playback method according to an embodiment of the present application. As shown in fig. 1, the data playback method in the embodiment of the present application mainly includes the following steps 101 to 108.

Step 101, intelligent equipment responds to a data processing request, and obtains intelligent equipment input data based on sensing data acquired by a sensor of the intelligent equipment;

in a specific implementation process, different sensors are usually set in the intelligent device to sense surrounding scenes and obtain sensing data. As the sensor may comprise at least one of a lidar, a millimeter wave radar, a camera device, and the acquired perception data may comprise at least one of image data, lidar data or millimeter wave radar data.

Taking an intelligent device as an example of an autonomous vehicle, fig. 2 is a schematic structural diagram of a sensing system in the autonomous vehicle. As shown in fig. 2, the sensing system is responsible for identifying target elements such as dynamic and static targets around the vehicle body. The input of the camera is generally a camera device (such as a camera), a laser radar, a millimeter wave radar and the like, and in order to cover the 360-degree range of a vehicle body, the camera generally comprises a front side camera (front left side and front right side), a rear side camera (rear left side and rear right side), a front view camera and a rear view camera. Corresponding image data can be acquired by utilizing each camera, corresponding laser radar data can be acquired by utilizing a laser radar, and corresponding millimeter wave radar data can be acquired by utilizing a millimeter wave radar.

In a specific implementation process, laser radar data is sent into a laser radar detection model, and a single-frame laser radar target is output. And sending the images of the cameras at the same moment into a visual detection model, and outputting a single-frame visual target. And millimeter wave radar sensing is generally carried out, and a single frame millimeter wave radar target is output by a related sensing algorithm. The laser radar target, the visual target and the millimeter wave radar target at the same moment are input into a post-processing fusion model to obtain a perception target, and the information such as the 3D position, the speed, the acceleration, the angle, the angular velocity and the like of the target is output by multi-sensor and multi-frame fusion.

Therefore, when the data processing request is received and responded, the data processing operation to be executed can be analyzed from the data processing request, and then the intelligent device input data can be obtained based on the sensing data acquired by the sensor of the intelligent device.

Specifically, referring to fig. 2, if the data processing operation to be performed is to detect a single frame visual target by the visual detection model, all image data may be input as smart device input data. If the data processing operation to be executed is to perform target fusion by the post-processing fusion model, the corresponding single-frame visual target, single-frame laser radar target and single-frame meter wave radar target can be obtained as intelligent equipment input data based on the image data, the laser radar data and the millimeter wave radar data respectively. And are not illustrated herein.

102, the intelligent device performs data processing on the intelligent device input data to obtain intelligent device output data and time sequence control information corresponding to the intelligent device output data and the intelligent device input data;

in a specific implementation process, after the input data of the intelligent device is obtained, the intelligent device can further perform data processing on the input data of the intelligent device, so that output data can be obtained, and time sequence control information corresponding to the output data of the intelligent device and the input data of the intelligent device can be recorded. The timing control information may include a timestamp of the smart device input data and a timestamp of the smart device output data.

Specifically, referring to fig. 2, as can be seen from the sensing system structure of fig. 2, when the automatic driving vehicle performs data processing on input data of a certain intelligent device, the following two pieces of timing control information can be recorded:

the first time sequence control information takes the visual detection model as an execution subject, and records the time sequence control information of the data processing, and the corresponding information is as follows:

visual detection of- # vision_detection

input_data{

{ front_left,0 μs } left front camera

{ front_right,0 μs } right front camera

{ front,0 μs } front view camera

{ rear_left,0 μs } left rear camera

{ rear_right,0 μs } right rear camera

{ rear, o [ mu ] s } rearview camera

process_info{

start_time;2 mus, start time

finish_time;3 mus, completion time

time=1 } execution order

output_data { version_object, 0 μs } visual object

The second time sequence control information takes the post-processing fusion model as an execution main body, and records the time sequence control information of the data processing, and the corresponding information is as follows:

post process post processing

input_data{

{ vision_object,0 μs } visual object

{ radar_object,0 μs } radar target

{ lidar_object,0 μs } lidar target

process_info{

start_time;5 mus, start time

finish_time;7 mus, completion time

time=1 } execution order

output_data { { { version_object, 0 μs } } sense target

As can be seen from the programs based on the two timing control information, each timing control information may include process information, a timestamp of the input data of the intelligent device, a timestamp of the output description information and the output data of the intelligent device, and so on. The time stamp of the input data of the intelligent device in the first time sequence control information can be 0 mu s of the time stamp of the input image data of the left front side camera, etc. The timestamp of the output data of the device may be 0 μs of the output_data timestamp in the first timing control information. The process information may include a start time, a completion time, and an execution order, and details of the information corresponding to the first timing control information are not described herein.

In a specific implementation process, because the cameras are independently exposed and independently transmitted, the acquired images arrive at the visual detection model at different moments, the images acquired by exposure at the same moment are required to be formed into an image set and then are unified to the model for detection. The single-frame visual target, the single-frame laser radar target and the single-frame millimeter radar target which are detected at the same time form a multi-mode data set and then are uniformly delivered to a post-processing fusion model to perform state estimation on the targets, and the sequence among frames, namely the inter-frame sequencing information, is ensured. The inter-frame ordering information comprises first inter-frame ordering information of a plurality of sensing data from the same type of sensor when the sensing data are processed to obtain the sensing result of the type, and/or second inter-frame ordering information of a plurality of sensing results when the sensing results of different types of sensors are fused to obtain the fused sensing result. For example, the ordering information of the perceived data between different cameras may be used as the first inter-frame ordering information, referring to fig. 2, and the first inter-frame ordering information illustrated in fig. 2 may be an ordering sequence of an image data frame of a left front camera, an image data frame of a right front camera, an image data frame of a front view camera, an image data frame of a left rear camera, an image data frame of a right rear camera, and an image data frame of a rear view camera. The ordering information of the sensing results of the different sensors may be used as the second inter-frame ordering information. Referring to fig. 2, the second inter-frame ordering information illustrated in fig. 2 may be ordering information of a data frame corresponding to a visual target, a data frame corresponding to a millimeter radar target, and a data frame corresponding to a laser radar target.

Based on the two time sequence control information, the data input by the intelligent equipment can be traced back through the time sequence control information, and then the data related to the intelligent equipment can be traced back through the data input by the intelligent equipment until all the data on the processing link at the time are found. For example, the input data of the intelligent device corresponding to the second time sequence control information is a visual target, and the input data of the intelligent device corresponding to the first time sequence control information is output data (visual target).

For further explanation of the traceability of all data on the link, reference can be made to fig. 3, fig. 3 being a data flow dependency graph. The plurality of task programs of the smart device may include a related program of the trace record object 1 to a related program of the trace record object 6. After the task is executed, the input data and the output data corresponding to each tracking record object can be determined according to the obtained time sequence control information, and then the data flow dependence is obtained.

Specifically, the output data of the trace record object 1 is a1, the output data of the trace record object 1 is b1, the output data of the trace record object 3 is c2, the output data of the trace record object 4 is y1 and z1, the input data are a1, b1 and c2, the output data of the trace record object 5 is x1, the input data are y1 and c2, the output data of the trace record object 6 is in1 and in2, and the input data is out1. As can be seen from fig. 3, the data flow is dependent on y1, c2 for the output data x1, and y1 is again dependent on a1, b1, c2.

Step 103, the intelligent device sends the intelligent device input data, the intelligent device output data and the time sequence control information to a server;

in a specific implementation process, after each data processing request is executed by the intelligent device, the intelligent device input data, the intelligent device output data and the recorded time sequence control information are sent to the server, so that the server can complete data playback based on the intelligent device input data, the intelligent device output data and the recorded time sequence control information, obtain server data to be played back, which is consistent with the timestamp of the intelligent device output data, and play back the server data to be played back based on the timestamp of the server output data.

104, the server responds to the data playback request to acquire time sequence control information;

in a specific implementation process, after receiving the intelligent device input data, the intelligent device output data and the recorded time sequence control information, the server can load the intelligent device input data and the intelligent device output data into a data stream buffer and load the time sequence control information into a control stream buffer. When the server receives the data playback request, timing control information may be then acquired from the control stream buffer in response to the data playback request.

In a specific implementation process, the timing control information further includes the smart device process information described above. When there is timing control information to be acquired in the control flow buffer, the timing control information can be directly acquired. When a plurality of pieces of time sequence control information to be acquired exist in the control flow buffer, determining the acquisition sequence of the plurality of pieces of time sequence control information to be acquired based on the intelligent equipment process information corresponding to the plurality of pieces of time sequence control information to be acquired; and sequentially acquiring one of the plurality of timing control information to be acquired as the timing control information based on the acquisition order.

In a specific implementation process, the acquisition order of the plurality of timing control information to be acquired may be determined according to the following three manners:

first kind: and if the plurality of time sequence control information to be acquired are all acquired based on the same type of equipment data processing operation, determining the acquisition sequence according to the execution sequence in the intelligent equipment process information and the sequence from small to large.

Specifically, the device data processing operation includes an operation of performing data processing on sensing data from the same type of sensor to obtain a sensing result of the type of sensor, that is, an operation of performing data processing on image data obtained from a camera by using a visual detection model corresponding to the first moment control information to obtain a visual target. Or, the device data processing operation may include an operation of fusing sensing results of different types of sensors to obtain a fused sensing result, that is, an operation of performing data processing on different sensing results by using a post-processing fusion model pair corresponding to the second time control information to obtain a fused sensing target.

In a specific implementation process, each time a certain model in the intelligent device is subjected to data processing operation, 1 is added to time in process information of the intelligent device corresponding to the model, and the sequence of executing the data processing operation of the device each time is obtained.

Therefore, if a certain model in the intelligent device executes a plurality of device data processing operations, a plurality of obtained time sequence control information are all obtained based on the same type of device data processing operation, and the acquisition sequence can be determined according to the execution sequence in the process information and the sequence from small to large. That is, when time=1, the acquisition order is the forefront, when time=2, the acquisition order is the rear, and so on, the acquisition orders of the plurality of timing control information are obtained.

Second kind: if the plurality of time sequence control information to be acquired are respectively acquired based on different types of equipment data processing operations, determining the acquisition sequence according to the starting time in the intelligent equipment process information and the sequence from front to back of the time;

specifically, referring to the two timing control information described above, two different kinds of device data processing operations are respectively performed. At this time, it is necessary to assign a corresponding time sequence to each time sequence control information according to the start time in the process information, so as to obtain the acquisition order of the plurality of time sequence control information. The start time of the first time sequence control information is 2 mu s, and the second time sequence control information is 5 mu s, so that the acquisition sequence of the first time sequence control information is earlier, and the acquisition sequence of the second time sequence control information is later.

Third kind: if a part of the time sequence control information to be acquired in the plurality of time sequence control information to be acquired is acquired based on the same type of equipment data processing operation, and the rest of the time sequence control information to be acquired is acquired based on different types of equipment data processing operation respectively, determining the acquisition order according to the starting time and the execution order in the intelligent equipment process information. That is, the first and second cases are included at the same time, and the acquisition order of playing back the plurality of time sequence control information is obtained according to the first and second modes at this time, which is not described herein.

As can be seen from the foregoing description, when the intelligent device performs the device data processing operation, the inter-frame ordering information exists between the intelligent device input data, so, in order to further ensure the consistency of the server output data and the intelligent device output data, a plurality of intelligent device input data corresponding to the time stamp and the inter-frame ordering information may be sequentially acquired according to the time stamp of the intelligent device input data and the inter-frame ordering information.

Step 105, the server obtains intelligent equipment input data corresponding to the time sequence control information based on the time sequence control information;

In a specific implementation process, after the server acquires the timing control information, the server may acquire the intelligent device input data corresponding to the timing control information. And acquiring intelligent device input data corresponding to the time stamp of the intelligent device input data in the time sequence control information. For example, the data stream buffer may include input data of a certain camera under different time stamps, and the time sequence control information describes that the data of the camera at 0 μs is the target data, and the input data of the camera at 0 μs in the data information base is not acquired any more, so that after traversing the data information base, the intelligent device input data corresponding to the time sequence control information can be obtained.

Step 106, the server executes server data processing operation based on the intelligent device input data to obtain server output data;

in a specific implementation process, after obtaining the input data of the intelligent device, the server can call the corresponding server data processing operation and execute the operation, so that the output data of the server can be obtained.

Step 107, adding a timestamp of the server output data to the server output data based on the timestamp of the intelligent device output data to obtain server to-be-played-back data consistent with the timestamp of the intelligent device output data;

In a specific implementation process, the server may add the timestamp of the server output data to the server output data based on the timestamp of the intelligent device output data, so that the timestamp of the server output data is consistent with the timestamp of the intelligent device output data, and the server output data added with the timestamp may be used as the data to be played back by the server, so that when the same intelligent device input data is required to be played back later, the server may directly play back the data to be played back by the server based on the timestamp of the server output data.

In a specific implementation process, the data to be played back by the server may also be stored in the data information base, so as to be used for input data of other server data processing operations.

And step 108, the server plays back the data to be played back of the server based on the time stamp of the output data of the server.

According to the data playback method, a server responds to a data playback request to acquire time sequence control information, and intelligent equipment input data corresponding to the time sequence control information is acquired based on the time sequence control information; based on the intelligent equipment input data, executing server data processing operation to obtain server output data; adding a time stamp of the server output data to the server output data based on the time stamp of the intelligent device output data to obtain server to-be-played-back data consistent with the time stamp of the intelligent device output data; based on the timestamp of the output data of the server, the data to be played back of the server is played back, so that the server plays back according to the time sequence of each data operation executed by the intelligent device, the output data of the server is consistent with the output data of the intelligent device, and the corresponding adjustment strategy is conveniently and accurately made for the automatic driving system.

Fig. 4 is a flowchart illustrating a specific implementation of a data playback method according to an embodiment of the present application. As shown in fig. 4, the data playback method is only executed in a server, specifically, the timing control information and the data information may be loaded into a control stream buffer and a data stream buffer, respectively, where the data information may include intelligent device input data, intelligent device output data, and the like; determining the acquisition sequence of the time sequence control information to be acquired according to the time series and/or the starting time in the process information of each time sequence control information to be acquired so as to acquire each piece of time sequence control information in the control flow buffer in sequence; detecting whether timing control information exists; if not, ending; if yes, analyzing the time stamp of the input data of the intelligent equipment in the time sequence control information, taking out the data corresponding to the time stamp from the data stream buffer to form an input data set, inputting the input data set to a program corresponding to the data processing operation of the corresponding server, and running the program; and stamping the obtained server output data with a timestamp of the intelligent device output data to obtain the data to be played back of the server, and recording the data to be played back of the server into a data stream buffer for the downstream task to take as input. The cycle is repeated until all timing control information is processed.

It should be noted that, although the foregoing embodiments describe the steps in a specific sequential order, it should be understood by those skilled in the art that, in order to achieve the effects of the present application, different steps need not be performed in such an order, and may be performed simultaneously (in parallel) or in other orders, and these variations are within the scope of protection of the present application.

It will be appreciated by those skilled in the art that the present application may implement all or part of the above-described methods according to the above-described embodiments, or may be implemented by means of a computer program for instructing relevant hardware, where the computer program may be stored in a computer readable storage medium, and the computer program may implement the steps of the above-described embodiments of the method when executed by a processor. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable storage medium may include: any entity or device, medium, usb disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier wave signals, telecommunications signals, software distribution media, and the like capable of carrying the computer program code. It should be noted that the computer readable storage medium may include content that is subject to appropriate increases and decreases as required by jurisdictions and by jurisdictions in which such computer readable storage medium does not include electrical carrier signals and telecommunications signals.

Further, the application also provides a data playback device.

Referring to fig. 5, fig. 5 is a main structural block diagram of a data playback apparatus according to an embodiment of the present application. As shown in fig. 5, the data playback device in the embodiment of the present application may include a response module 51, an acquisition module 52, and a playback module 53.

A response module 51 for acquiring timing control information in response to a data playback request; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment;

an obtaining module 52, configured to obtain, based on the timing control information, intelligent device input data corresponding to the timing control information; the intelligent equipment input data is obtained based on sensing data acquired by a sensor of the intelligent equipment; the perception data comprises at least one of image data, laser radar data or millimeter wave radar data;

a playback module 53, configured to add a timestamp of the server output data to the server output data based on the timestamp of the intelligent device output data, so as to obtain server data to be played back, where the server data is consistent with the timestamp of the intelligent device output data; and playing back the data to be played back of the server based on the timestamp of the output data of the server.

The technical principles of the two embodiments of the data playback apparatus and the technical problems to be solved and the technical effects to be produced are similar, and those skilled in the art can clearly understand that, for convenience and brevity of description, the specific working process and the related description of the data playback apparatus may refer to the description of the embodiments of the data playback method, and will not be repeated herein.

Further, the application also provides a server.

Referring to fig. 6, fig. 6 is a main structural block diagram of a server according to an embodiment of the present application. As shown in fig. 6, the server in the embodiment of the present application may include a processor 61 and a memory 62.

The memory 62 may be configured to store a program for executing the data playback method of the above-described method embodiment, and the processor 61 may be configured to execute the program in the memory 62, including but not limited to a program for executing the data playback method of the above-described method embodiment. For convenience of explanation, only those portions relevant to the embodiments of the present application are shown, and specific technical details are not disclosed, refer to the method portions of the embodiments of the present application. The smart device may be a control device formed from a variety of electronic devices.

In one implementation, the number of memory 62 and processor 61 may be multiple. And a program for executing the data playback method of the above-described method embodiment may be divided into a plurality of sub-programs, each of which may be loaded and executed by the processor 61 to perform the different steps of the data playback method of the above-described method embodiment, respectively. Specifically, each of the sub-programs may be stored in a different memory 62, respectively, and each of the processors 61 may be configured to execute the programs in one or more memories 62 to collectively implement the data playback method of the above-described method embodiment, that is, each of the processors 61 executes the different steps of the data playback method of the above-described method embodiment, respectively, to collectively implement the data playback method of the above-described method embodiment.

The plurality of processors 61 may be processors disposed on the same device, for example, the device may be a high-performance device composed of a plurality of processors, and the plurality of processors 61 may be processors configured on the high-performance device. The plurality of processors 61 may be processors disposed on different devices, for example, the devices may be a server cluster, and the plurality of processors 61 may be processors on different servers in the server cluster.

Further, the application also provides intelligent equipment.

Referring to fig. 7, fig. 7 is a main structural block diagram of a server according to an embodiment of the present application. As shown in fig. 7, the server in the embodiment of the present application may include a processor 71 and a memory 72.

The memory 72 may be configured to store a program for executing the data playback method of the above-described method embodiment, and the processor 71 may be configured to execute the program in the memory 72, including, but not limited to, the program for executing the data playback method of the above-described method embodiment. For convenience of explanation, only those portions relevant to the embodiments of the present application are shown, and specific technical details are not disclosed, refer to the method portions of the embodiments of the present application. The smart device may be a control device formed from a variety of electronic devices.

In one implementation, the number of memory 72 and processors 71 may each be multiple. And a program for executing the data playback method of the above-described method embodiment may be divided into a plurality of sub-programs, each of which may be loaded and executed by the processor 71 to perform the different steps of the data playback method of the above-described method embodiment, respectively. Specifically, each of the sub-programs may be stored in a different memory 72, respectively, and each of the processors 71 may be configured to execute the programs in one or more memories 72 to collectively implement the data playback method of the above-described method embodiment, that is, each of the processors 71 executes the different steps of the data playback method of the above-described method embodiment, respectively, to collectively implement the data playback method of the above-described method embodiment.

The plurality of processors 71 may be processors disposed on the same device, for example, the device may be a high-performance device composed of a plurality of processors, and the plurality of processors 71 may be processors disposed on the high-performance device. The plurality of processors 71 may be processors disposed on different devices, for example, the devices may be a server cluster, and the plurality of processors 71 may be processors on different servers in the server cluster.

Further, the application also provides a data playback system.

Referring to fig. 8, fig. 8 is a main structural block diagram of a data playback system according to an embodiment of the present application. As shown in fig. 8, the server in the embodiment of the present application may include a smart device 81 and a server 82.

The intelligent device 81 is configured to record timing control information of any task when the task is scheduled, and send the timing control information to the server;

the server 82 is configured to perform the data playback method of the above-described embodiment.

Further, the present application also provides a computer-readable storage medium. In one computer-readable storage medium embodiment according to the present application, the computer-readable storage medium may be configured to store a program that performs the data playback method of the above-described method embodiment, the program being loadable and executable by a processor to implement the above-described data playback method. For convenience of explanation, only those portions relevant to the embodiments of the present application are shown, and specific technical details are not disclosed, refer to the method portions of the embodiments of the present application. The computer readable storage medium may be a memory device formed by including various electronic devices, and optionally, in embodiments of the present application, the computer readable storage medium is a non-transitory computer readable storage medium.

Further, it should be understood that, since the respective modules are merely set to illustrate the functional units of the apparatus of the present application, the physical devices corresponding to the modules may be the processor itself, or a part of software in the processor, a part of hardware, or a part of a combination of software and hardware. Accordingly, the number of individual modules in the figures is merely illustrative.

Those skilled in the art will appreciate that the various modules in the apparatus may be adaptively split or combined. Such splitting or combining of specific modules does not lead to a deviation of the technical solution from the principles of the present application, and therefore, the technical solution after splitting or combining will fall within the protection scope of the present application.

It should be noted that, the personal information of the relevant user possibly related to each embodiment of the present application is personal information that is strictly according to requirements of laws and regulations, follows legal, legal and necessary principles, and processes the personal information actively provided by the user or generated by using the product/service in the process of using the product/service based on the reasonable purpose of the business scenario, and is obtained by the user through authorization.

The personal information of the user processed by the application may be different according to the specific product/service scene, and the specific scene that the user uses the product/service may be referred to as account information, equipment information, driving information, vehicle information or other related information of the user. The present application treats the user's personal information and its processing with a high diligence.

The method and the device have the advantages that safety of personal information of the user is very important, and safety protection measures which meet industry standards and are reasonable and feasible are adopted to protect the information of the user and prevent the personal information from unauthorized access, disclosure, use, modification, damage or loss.

Thus far, the technical solutions of the present application have been described with reference to the embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of protection of the present application is not limited to these specific embodiments. Equivalent modifications and substitutions for related technical features may be made by those skilled in the art without departing from the principles of the present application, and such modifications and substitutions will be within the scope of the present application.

Claims (10)

1. A data playback method, characterized by being applied to a server, comprising:

responding to a data playback request, and acquiring time sequence control information; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment;

acquiring intelligent equipment input data corresponding to the time sequence control information based on the time sequence control information; the intelligent equipment input data is obtained based on sensing data acquired by a sensor of the intelligent equipment; the perception data comprises at least one of image data, laser radar data or millimeter wave radar data;

Based on the intelligent equipment input data, executing server data processing operation to obtain server output data;

adding a time stamp of the server output data to the server output data based on the time stamp of the intelligent device output data to obtain server to-be-played-back data consistent with the time stamp of the intelligent device output data;

and playing back the data to be played back of the server based on the timestamp of the output data of the server.

2. The data playback method of claim 1, wherein the timing control information further comprises smart device process information;

the method further comprises the steps of:

if there are a plurality of timing control information to be acquired, the acquiring the timing control information includes:

determining the acquisition sequence of the time sequence control information to be acquired based on the intelligent equipment process information corresponding to each of the time sequence control information to be acquired;

and sequentially acquiring one of the plurality of timing control information to be acquired as the timing control information based on the acquisition order.

3. The data playback method of claim 2, wherein determining the acquisition order of the plurality of timing control information based on the smart device process information to which the plurality of timing control information to be acquired corresponds, respectively, comprises:

If the plurality of time sequence control information to be acquired are all acquired based on the same type of equipment data processing operation, determining the acquisition sequence according to the execution sequence in the intelligent equipment process information and the sequence from small to large;

the device data processing operation comprises the operation of performing data processing on sensing data from the same type of sensor to obtain sensing results of the type of sensor, or the operation of fusing sensing results of different types of sensors to obtain fused sensing results.

4. The data playback method of claim 2, wherein determining the acquisition order of the plurality of timing control information based on the smart device process information to which the plurality of timing control information to be acquired corresponds, respectively, comprises:

if the plurality of time sequence control information to be acquired are respectively acquired based on different types of equipment data processing operations, determining the acquisition sequence according to the starting time in the intelligent equipment process information and the sequence from front to back of the time;

the device data processing operation comprises the operation of performing data processing on sensing data from the same type of sensor to obtain sensing results of the type of sensor, or the operation of fusing sensing results of different types of sensors to obtain fused sensing results.

5. The data playback method of claim 1, wherein determining an acquisition order of the plurality of timing control information based on the smart device process information to which the plurality of timing control information to be acquired corresponds, comprises:

if a part of the time sequence control information to be acquired in the plurality of time sequence control information to be acquired is acquired based on the same type of equipment data processing operation, and the rest of the time sequence control information to be acquired is acquired based on different types of equipment data processing operation respectively, determining the acquisition order according to the starting time and the execution order in the intelligent equipment process information;

the device data processing operation comprises the operation of performing data processing on sensing data from the same type of sensor to obtain sensing results of the type of sensor, or the operation of fusing sensing results of different types of sensors to obtain fused sensing results.

6. The data playback method as recited in claim 1, further comprising:

and storing the data to be played back of the server to the data information base.

7. The method according to claim 1, wherein the timing control information further includes inter-frame ordering information, the inter-frame ordering information includes first inter-frame ordering information of a plurality of sensing data from a same type of sensor when the sensing result is obtained by data processing, and/or second inter-frame ordering information of a plurality of sensing results when the sensing results of different types of sensors are fused to obtain a fused sensing result;

Based on the timing control information, acquiring the intelligent device input data includes:

and sequentially acquiring a plurality of intelligent device input data corresponding to the time stamp and the inter-frame ordering information according to the time stamp of the intelligent device input data and the inter-frame ordering information.

8. A data playback method, characterized by being applied to a smart device, the smart device including a sensor, the method comprising:

responding to a data processing request, and obtaining intelligent equipment input data based on sensing data acquired by a sensor of the intelligent equipment, wherein the sensing data comprises at least one of image data, laser radar data or millimeter wave radar data;

performing data processing on the intelligent device input data to obtain intelligent device output data, and timing control information corresponding to the intelligent device output data and the intelligent device input data; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment;

and sending the intelligent device input data, the intelligent device output data and the time sequence control information to a server, so that when the server responds to a data playback request, obtaining the server to-be-played-back data consistent with the time stamp of the intelligent device output data based on the acquired time sequence control information, and playing back the server to-be-played-back data based on the time stamp of the server output data.

9. A data playback method, applied to a data playback system including an intelligent device and a server, wherein information interaction is performed between the intelligent device and the server, the method comprising:

the intelligent equipment responds to the data processing request, and obtains intelligent equipment input data based on sensing data acquired by a sensor of the intelligent equipment; performing data processing on the intelligent device input data to obtain intelligent device output data, and timing control information corresponding to the intelligent device output data and the intelligent device input data; transmitting the intelligent device input data, the intelligent device output data and the time sequence control information to a server; wherein the perception data comprises at least one of image data, laser radar data or millimeter wave radar data; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment;

the server responds to the data playback request and acquires time sequence control information; acquiring intelligent equipment input data corresponding to the time sequence control information based on the time sequence control information; acquiring intelligent equipment input data corresponding to the time sequence control information based on the time sequence control information; adding a time stamp of the server output data to the server output data based on the time stamp of the intelligent device output data to obtain server to-be-played-back data consistent with the time stamp of the intelligent device output data; based on the timestamp of the output data of the server, playing back the data to be played back of the server; the time sequence control information comprises a time stamp of input data of the intelligent equipment and a time stamp of output data of the intelligent equipment; the intelligent equipment input data is obtained based on sensing data acquired by a sensor of the intelligent equipment; the perception data includes at least one of image data, laser radar data, or millimeter wave radar data.

10. A server comprising a processor and a memory, the memory being adapted to store a plurality of program codes, the program codes being adapted to be loaded and executed by the processor to perform the data playback method of any one of claims 1 to 7.