WO2022048387A1

WO2022048387A1 - Data storage method and system, and data calling method and system

Info

Publication number: WO2022048387A1
Application number: PCT/CN2021/110847
Authority: WO
Inventors: 闵令昂
Original assignee: 北京航迹科技有限公司
Priority date: 2020-09-07
Filing date: 2021-08-05
Publication date: 2022-03-10
Also published as: CN112069368A

Abstract

A data storage method and system, and a data calling method and system. The data storage method comprises: obtaining an original data set (410), the original data set comprising a plurality of data elements, and each data element having type information marking the type of the data element; obtaining the number N of different types according to the type information, and establishing N different target data sets (420); and storing, on the basis of the type information of the data elements in the original data set and each target data set, in corresponding target data sets the data elements corresponding to the target data sets (430), the target data sets being stored in a first storage device. The data calling method comprises: obtaining a data calling request sent by a user end (710), the data calling request comprising at least the type of data to be called; obtaining part of the data in a target data set on the basis of the data calling request to obtain the data to be called (720); and sending said data to a second storage device of the user end (730).

Description

Data storage, calling method and system

cross reference

This application claims the priority of Chinese patent application 202010931768.6 filed on September 7, 2020, the contents of which are fully incorporated herein by reference.

technical field

The present application relates to the field of information technology, and in particular, to a data storage and invocation method and system.

Background technique

In road testing, autonomous driving test vehicles usually need to collect a large amount of data (referred to as "road test data") for analysis, debugging and other purposes. The relevant personnel's demand for drive test data is usually targeted, that is, only part of the large amount of data needs to be called. For example, a specific team usually only needs to call drive test data belonging to a specific type rather than all types of drive test data (eg, an image processing team only needs drive test data related to image types). For another example, in some cases, only drive test data within a specific range (for example, a specific time period) needs to be called.

Therefore, it is desirable to provide a data storage and/or recall solution that can efficiently recall part of the data that meets the specific needs of users from a huge amount of data.

SUMMARY OF THE INVENTION

One aspect of the present application provides a data storage method executed by a computing device, characterized in that the method includes: acquiring an original data set, the original data set includes a plurality of data elements, each data element has a label for the data The type information of the meta type; according to the type information of the data elements in the original data set, the number N of different types is obtained, and N different target data sets are established correspondingly, and the N different target data sets are associated with different types of data sets. corresponding to the data element; wherein, N is an integer greater than or equal to 2; and based on the type information of the data element in the original data set and each target data set, the data element corresponding to the target data set is stored in the corresponding target data set In the data set, the target data set is stored in the first storage device.

In some embodiments, the dataset is a file, and the data elements of the file are messages.

In some embodiments, the types include one or more of an image class, a location class, a sensor class, a packet class, and a controller area network bus class.

In some embodiments, the method further includes: establishing index information of the target data set, where the index information at least includes meta-identification information and storage location information corresponding to each data element in the target data set; wherein , and the meta identification information refers to the identification information of the corresponding data element.

In some embodiments, the data elements in the target data set are arranged in chronological order, and the meta identification information includes time information of the corresponding data elements.

In some embodiments, the index information further includes set identification information of the original data set corresponding to each data element in the target data set; wherein, the set identification information refers to the identification information of the original data set.

In some embodiments, the data in the raw data set includes data generated or collected during operation of the autonomous vehicle.

In some embodiments, the method further includes: receiving a data call request sent by the client, the data call request at least including the type of the data to be called; based on the data call request from the N different targets The data set determines a corresponding type of target data set; obtains the data to be called based on the data elements in the determined target data set; and sends the data to be called to the second storage device of the client.

In some embodiments, the obtaining the data to be called based on the data elements in the determined target data set further comprises: acquiring the target data set and the data elements stored therein from the first storage device; Dividing the target data set into multiple target data subsets at preset time intervals; and acquiring, based on the data call request, data elements corresponding to some target data subsets in the multiple target data subsets, the The data to be called includes the data elements corresponding to the partial target data subsets.

In some embodiments, the obtaining the data to be called based on the data elements in the determined target data set further comprises: converting the determined target data set acquired in the first storage device and the stored data thereof. The data element is sent to a third storage device, and the first storage device is farther from the user end than the third storage device is far from the user end; the target data set is divided into multiple sets at preset time intervals The target data subset is stored in the third storage device; multiple logical files are established, each logical file corresponds to one of the multiple target data subsets, and each logical file includes the target data subsets. index information corresponding to the data element; and based on the data call request and the logical file, obtain, from the third storage device, data elements stored in some target data subsets in the plurality of target data subsets, the The data to be called includes the data elements corresponding to the partial target data subsets.

Another aspect of the present application provides a data storage system, characterized in that, the system includes an original data set acquisition module, a target data set establishment module and a storage module. The original data set acquisition module is used for acquiring an original data set, the original data set includes a plurality of data elements, and each data element has type information marking the type of the data element. The target data set establishment module is used to obtain the number N of different types according to the type information of the data elements in the original data set, and correspondingly establish N different target data sets, the N different target data sets and Different types of data elements correspond; among them, N is an integer greater than or equal to 2. The storage module is configured to store the data elements corresponding to the target data set in the corresponding target data set based on the type information of the data elements in the original data set and the target data set.

In some embodiments, the system further includes an index information establishment module, configured to establish index information of the target data set, where the index information at least includes meta-identification information of one-to-one correspondence of each data element in the target data set and storage location information; wherein, the meta identification information refers to the identification information of the corresponding data element.

Another aspect of the present application provides a storage medium, wherein the storage medium is used for storing computer instructions, and after the computer reads the computer instructions in the storage medium, the above data storage method is executed.

Yet another aspect of the present application provides a data calling method executed by a computing device, wherein the data elements in the original data set are stored in a corresponding target data set according to the above data storage method, and the target data set is stored in a computer in the first storage device associated with the apparatus. The data invocation method includes: acquiring a data invocation request sent by a client, where the data invocation request at least includes the type of data to be invoked; acquiring part of the data in the target data set based on the data invocation request to obtain the data invoking Data to be called, the partial data includes data elements in the target data set corresponding to the type to which the data to be called belongs; and the data to be called is sent to the second storage device of the client.

In some embodiments, the target data set has corresponding index information, and the index information at least includes meta-identification information and storage location information corresponding to each data element in the target data set, wherein the meta-identification information refers to The identification information of the corresponding data element; the data call request further includes the meta qualification related to the meta identification information. The acquiring, based on the data calling request, the data elements in the target data set of the corresponding type includes: acquiring, based on the data calling request, index information corresponding to the corresponding type and satisfying the element qualification; and based on the acquired index information The storage location of the data element is obtained.

In some embodiments, the data elements in the target data set are arranged in chronological order, the meta identification information includes time information of the corresponding data elements, and the meta qualification includes a time range corresponding to the data to be called.

In some embodiments, the acquiring part of the data in the target data set based on the data calling request further includes: dividing the target data set into multiple target data subsets at preset time intervals; The data invocation request is used to obtain the data elements corresponding to some target data subsets in the target data subsets, and the data to be invoked includes the data elements corresponding to the partial target data subsets.

In some embodiments, the acquiring part of the data in the target data set based on the data invocation request further includes: sending the target data set acquired in the first storage device and the data elements stored therein to the first storage device. three storage devices, the first storage device is farther from the user terminal than the third storage device is far from the user terminal; the target data set is divided into multiple target data subsets at preset time intervals and storing in the third storage device; establishing multiple logical files, each logical file corresponding to one of the multiple target data subsets, the logical file including index information corresponding to the data elements in the target data subset; and based on For the data calling request and the logic file, data elements corresponding to some target data subsets in the plurality of target data subsets are obtained from the third storage device, and the data to be called includes the partial target data The data element corresponding to the subset.

In some embodiments, the index information further includes set identification information of the original data set corresponding to each data element in the target data set, wherein the set identification information refers to the identification information of the original data set; the data invocation request further includes Includes set qualifications related to set identification information. The acquiring index information corresponding to the corresponding type and satisfying the meta-qualifying condition based on the data calling request includes: acquiring index information corresponding to the corresponding type and satisfying the set-qualifying condition based on the data calling request.

Yet another aspect of the present application provides a data calling system, characterized in that the data elements in the original data set are stored in the corresponding target data set according to the above data storage method, and the target data set is stored in the first data set associated with the computer device. in a storage device. The data calling system includes a user request acquiring module and a calling module. The user request obtaining module is configured to obtain a data calling request sent by the client, where the data calling request at least includes the type of the data to be called. The calling module is configured to acquire, based on the data calling request, part of the data in the target data set to obtain the data to be called, where the partial data includes the data in the target data set corresponding to the type to which the data to be called belongs. data element.

In some embodiments, the target data set has corresponding index information, and the index information at least includes meta-identification information and storage location information corresponding to each data element in the target data set, wherein the meta-identification information is Refers to the identification information of the corresponding data element; the data invocation request also includes meta qualifications related to the meta identification information. The calling module includes an index information acquisition unit, a segment storage unit and a data element acquisition unit. The index information obtaining unit is configured to obtain index information corresponding to the corresponding type and satisfying the meta-qualification condition based on the data call request. The segment storage unit is configured to divide the target data set into a plurality of target data subsets at preset time intervals, and store each target data subset separately. The data element obtaining unit is configured to obtain the data element based on the storage location in the obtained index information.

In some embodiments, the data calling system further includes a synchronization module, and the synchronization module is configured to send the data to be called to the second storage device of the client.

In some embodiments, the index information further includes set identification information of the original data set corresponding to each data element in the target data set, wherein the set identification information refers to the identification information of the original data set; the data invocation request further includes Includes set qualifications related to set identification information. The index information obtaining unit is further configured to obtain, based on the data call request, index information corresponding to the corresponding type and satisfying the set qualification condition.

Another aspect of the present application provides a storage medium, wherein the storage medium is used for storing computer instructions, and after the computer reads the computer instructions in the storage medium, the above data calling method is executed.

Description of drawings

The present application will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These examples are not limiting, and in these examples, the same numbers refer to the same structures, wherein:

1 is a schematic diagram of an application scenario of a data processing system according to some embodiments of the present application;

Figure 2 is a block diagram of an exemplary processing device according to some embodiments of the present application;

3 is a block diagram of another exemplary processing device according to some embodiments of the present application;

FIG. 4 is an exemplary flowchart of a data storage method according to some embodiments of the present application;

5 is a schematic diagram of storing different types of data elements in an original data set in a corresponding target data set according to some embodiments of the present application;

6 is a schematic diagram of index information corresponding to a target dataset according to some embodiments of the present application;

FIG. 7 is an exemplary flowchart of a data calling method according to some embodiments of the present application;

8 is a schematic diagram of a data calling process according to some embodiments of the present application;

9 is a schematic diagram of a data calling scenario according to some embodiments of the present application;

FIG. 10 is a schematic diagram of data invocation according to some embodiments of the present application;

FIG. 11 is a schematic diagram of data storage and invocation according to some embodiments of the present application; and

FIG. 12 is a schematic diagram of a user interaction interface according to some embodiments of the present application.

detailed description

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that are used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application. For those of ordinary skill in the art, without any creative effort, the present application can also be applied to the present application according to these drawings. other similar situations. Unless obvious from the locale or otherwise specified, the same reference numbers in the figures represent the same structure or operation.

It should be understood that "system", "device", "unit" and/or "module" as used herein is a method used to distinguish different components, elements, parts, parts or assemblies at different levels. However, other words may be replaced by other expressions if they serve the same purpose.

As shown in this application and in the claims, unless the context clearly dictates otherwise, the words "a", "an", "an" and/or "the" are not intended to be specific in the singular and may include the plural. Generally speaking, the terms "comprising" and "comprising" only imply that the clearly identified steps and elements are included, and these steps and elements do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

Flow diagrams are used in this application to illustrate operations performed by a system according to an embodiment of the application. It should be understood that the preceding or following operations are not necessarily performed in the exact order. Instead, the various steps can be processed in reverse order or simultaneously. At the same time, other actions can be added to these procedures, or a step or steps can be removed from these procedures.

The embodiments of the present application can be applied to a data storage and invocation scenario where the amount of data is large and the user's demand for data is specific. In this scenario, the user usually only needs to invoke part of the data in the large amount of data. In some embodiments, the large amount of data may include road test data collected by an autonomous driving test vehicle during a road test. For example, in some application scenarios, the amount of road test data collected by the autonomous driving test vehicle can reach about 17MB/sec/vehicle, the average amount of road test data called each time can exceed 11G, and the average amount of road test data called every day It can exceed 8T, and the amount of data is large. The embodiments of the present application provide a data storage and/or calling method, which can efficiently call part of the data that meets the specific needs of the user from the large amount of data. It should be understood that the application scenarios of the data storage and invocation method and system of the present application are only some examples or embodiments of the present application. For those of ordinary skill in the art, without creative work, they can also The application is applied to other similar scenarios according to these figures. Although this application mainly takes the drive test data as an example for description, it should be noted that the principles of this application can also be applied to the storage and invocation of other data with a large amount of data and targeted data requirements of users, for example, Positioning data, production data, monitoring data, etc.

FIG. 1 is a schematic diagram of an application scenario of a data processing system according to some embodiments of the present application. In some embodiments, data processing system 100 may include a vehicle 110 (eg, vehicles 110-1, 110-2... and/or 110-n), server 120, terminal device 130, storage device 140, network 150, and Positioning and Navigation System 160 . The data processing system 100 can be applied to taxi services, security systems, network monitoring, driverless vehicles, and the like. It should be noted that the description about autonomous driving in this application is for illustration purposes only, and does not limit the scope of this application.

Vehicle 110 may be any type of autonomous vehicle, drone, or the like. An unmanned vehicle or drone can refer to a vehicle capable of achieving a certain level of driving automation. Exemplary levels of driving automation may include: a first level, where the vehicle is primarily supervised by humans and has certain autonomous functions (eg, autonomous steering or acceleration); a second level, where the vehicle has one or more brakes that can control the vehicle , advanced driver assistance systems (ADAS) for steering and/or acceleration (e.g., adaptive cruise control, lane keeping systems); Level 3, the vehicle is capable of driving itself when one or more specific conditions are met; Level 4, the vehicle Can operate without human input or supervision, but still subject to certain constraints (e.g., restricted to a certain area); Level 5, where the vehicle operates autonomously in all situations, etc., or any combination thereof. Vehicle 110 may also be a vehicle or other vehicle that is driven under human control for the purpose of collecting data.

In some embodiments, the vehicle 110 may have an equivalent structure that enables the vehicle 110 to move around or fly. For example, vehicle 110 may include conventional vehicle structures such as a chassis, suspension, steering (eg, steering wheel), braking (eg, brake pedal), accelerator, and the like. As another example, the vehicle 110 may have a body and at least one wheel. The body can be any body type, such as a sports car, coupe, sedan, pickup truck, station wagon, sport utility vehicle (SUV), minivan, or converted van. The at least one wheel may be all wheel drive (AWD), front wheel drive (FWR), rear wheel drive (RWD), or the like. In some embodiments, the vehicle 110 may be an electric vehicle, a fuel cell vehicle, a hybrid vehicle, a conventional internal combustion engine vehicle, or the like.

In some embodiments, the vehicle 110 can sense its environment and navigate using one or more detection units 112 . The detection unit 112 may include a global positioning system (GPS) module, a radar (eg, light detection and ranging (LiDAR)), an inertial measurement unit (IMU), a camera, etc., or any combination thereof. Radar (eg, LiDAR) can be used to scan the surrounding environment and generate point cloud data. The point cloud data can then be used to digitally 3D represent one or more objects around the vehicle 110 . A GPS module may refer to a device capable of receiving geographic location and time information from GPS satellites and calculating its geographic location. An IMU can refer to an electronic device that uses various inertial sensors to measure and provide the vehicle's specific force, angular velocity, and sometimes the magnetic field surrounding the vehicle. The various inertial sensors may include acceleration sensors (eg, piezoelectric sensors), velocity sensors (eg, Hall sensors), distance sensors (eg, radar, LIDAR, infrared sensors), rotational angle sensors (eg, tilt sensors), Traction related sensors (eg, force sensors). The camera may be configured to acquire one or more images related to objects within the camera range (eg, people, animals, trees, roadblocks, buildings, or vehicles).

In some embodiments, server 120 may be a single server or a group of servers. The server group may be centralized or distributed (eg, server 120 may be a distributed system). In some embodiments, server 120 may be local or remote. For example, server 120 may access information and/or data stored in terminal device 130 , detection unit 112 , vehicle 110 , storage device 140 , and/or positioning and navigation system 160 via network 150 . As another example, the server 120 may be directly connected to the terminal device 130 , the detection unit 112 , the vehicle 110 and/or the storage device 140 to access stored information and/or data. In some embodiments, server 120 may be implemented on a cloud platform or on-board computer. For example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distribution cloud, an internal cloud, a multi-layer cloud, etc., or any combination thereof. In some embodiments, server 120 may execute on a computing device that includes one or more components.

In some embodiments, server 120 may include processing device 122 . Processing device 122 may process information and/or data to perform one or more of the functions described herein. For example, the processing device 122 may establish a corresponding target data set according to the type information of the data elements in the original data set, and store the data elements corresponding to the target data set in the corresponding target data set. Further, the processing device 122 may store the target data set including the data elements in the storage device 140 or other storage device or system. As another example, the processing device 122 may create a query index for data stored in the storage device 140 or other storage devices or systems. Specifically, the raw data may include data generated by multiple vehicles during a drive test process, and may include camera data, radar data, and the like. The processing repository 122 may build a query index based on each vehicle's trip ID, time range, and type of data element in the raw data. In some embodiments, the processing device 122 may include one or more processing engines (eg, a single-chip processing engine or a multi-chip processing engine). For example only, the processing device 122 may include one or more hardware processors, such as a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical Arithmetic Processing Unit (PPU), Digital Signal Processor (DSP), Field Programmable Gate Array (FPGA), Programmable Logic Device (PLD), Controller, Microcontroller Unit, Reduced Instruction Set Computer (RISC), Microprocessor device, etc. or any combination thereof. In some embodiments, the processing device 122 may be integrated in the terminal device 130 .

In some embodiments, the terminal device 130 may include a mobile device 130-1, a tablet computer 130-2, a laptop computer 130-3, a vehicle built-in device 130-4, 130-5, etc., or any combination thereof. In some embodiments, the mobile device 130-1 may include a smart home device, a wearable device, a smart mobile device, a virtual reality device, an augmented reality device, or the like, or any combination thereof. In some embodiments, smart home devices may include smart lighting devices, control devices for smart appliances, smart monitoring devices, smart TVs, smart cameras, walkie-talkies, etc., or any combination thereof. In some embodiments, the wearable device may include a smart bracelet, smart footwear, smart glasses, smart helmets, smart watches, smart clothing, smart backpacks, smart accessories, etc., or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant (PDA), a gaming device, a navigation device, a POS device, etc., or any combination thereof. In some embodiments, the virtual reality device and/or the augmented reality device may include a virtual reality headset, virtual reality glasses, virtual reality goggles, augmented reality helmet, augmented reality glasses, augmented reality goggles, etc. or any combination thereof. For example, virtual reality devices and/or augmented reality devices may include Google ^™ Glasses, Oculus Rift, HoloLens, Gear VR, and the like. In some embodiments, vehicle built-in devices 130-4 may include an on-board computer, on-board television, and the like. In some embodiments, the server 120 may be integrated into the terminal device 130 . In some embodiments, the terminal device 130 may include a device with a positioning function to determine the location of the user and/or the terminal device 130 .

Storage device 140 may store data and/or instructions. In some embodiments, storage device 140 may store data obtained from vehicle 110 , detection unit 112 , processing device 122 , terminal device 130 , positioning and navigation system 160 , and/or external devices. For example, the storage device 140 may store drive test data obtained from the vehicle 110 . In some embodiments, storage device 140 may store data and/or instructions that may be executed or used to perform the example methods described in this application. For example, storage device 140 may store instructions that processing device 122 may execute to store and/or recall drive test data. In some embodiments, storage device 140 may include mass storage, removable storage, volatile read-write memory, read-only memory (ROM), the like, or any combination thereof. Exemplary mass storage may include magnetic disks, optical disks, solid state drives, and the like. Exemplary removable storage may include flash drives, floppy disks, optical disks, memory cards, magnetic disks, tapes, and the like. Exemplary volatile read-write memory may include random access memory (RAM). Exemplary RAMs may include dynamic random access memory (DRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), static random access memory (SRAM), thyristor random access memory (T-RAM), and zero capacitance Random Access Memory (Z-RAM), etc. Exemplary read-only memories may include masked read-only memory (MROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), Compact Disc Read Only Memory (CD-ROM) and Digital Versatile Disk Read Only Memory, etc. In some embodiments, the storage device 140 may further include a distributed file system (Hadoop Distributed File System, HDFS). In some embodiments, the distributed file systems may be located in different regions (eg, different countries, different regions, different sites, etc.) and associated with each other. Users can access the distributed file system in their region to obtain the data stored in it, and can also call the data in the distributed file systems in other regions through the distributed file system in their region. For example, the distributed file system may include a first distributed file system and a second distributed file system, the first distributed file system server belongs to the first region, and the second distributed file system server belongs to the second region. The drive test data is collected in the first area and stored in the first distributed system according to any method shown in the embodiments of this application. The location of the user terminal belongs to the second area, and the distance between the user terminal and the second distributed system server is smaller than the distance between the user terminal and the first distributed system server. In some embodiments, the user may invoke at least part of the data stored in the first distributed system through any method shown in the embodiments of this application. In some embodiments, the data processing system 100 can synchronize at least part of the data in the first distributed system with the second distributed system, and the user can use any method shown in the embodiments of this application to synchronize data from the second distributed system call this at least part of the data. In some embodiments, the storage device 140 may execute on a cloud platform. For example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distribution cloud, an internal cloud, a multi-layer cloud, etc., or any combination thereof.

In some embodiments, storage device 140 may be connected to network 150 for communication with one or more components in data processing system 100 (eg, server 120 , terminal device 130 , detection unit 112 , vehicle 110 , and/or positioning and navigation system 160 ). ) communication. One or more components in data processing system 100 may access data or instructions stored in storage device 140 via network 150 . In some embodiments, storage device 140 may be directly connected to one or more components in data processing system 100 (eg, server 120 , end device 130 , detection unit 112 , vehicle 110 , and/or positioning and navigation system 160 ) or with communication. In some embodiments, storage device 140 may be part of server 120 . In some embodiments, the storage device 140 may be integrated in the vehicle 110 .

Network 150 may facilitate the exchange of information and/or data. In some embodiments, one or more components in data processing system 100 (eg, server 120 , end device 130 , detection unit 112 , vehicle 110 , storage device 140 , and/or positioning and navigation system 160 ) /Send/obtain information and/or data from other components in data processing system 100. For example, processing device 122 may obtain drive test data from vehicle 110 via network 150 . For another example, the processing device 122 may obtain the data call request input by the user from the terminal device 130 via the network 150 . In some embodiments, the network 150 may be a wired network or a wireless network, or the like, or any combination thereof. By way of example only, the network 150 may include a cable network, a wired network, a fiber optic network, a telecommunications network, an internal network, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN) , Public Switched Telephone Network (PSTN), Bluetooth network, ZigBee network, Near Field Communication (NFC) network, etc. or any combination thereof. In some embodiments, network 150 may include one or more network access points. For example, network 150 may include wired or wireless network access points (eg, base stations and/or Internet exchange points 150-1, 150-2) through which one or more components of data processing system 100 may connect to network 150 to exchange data and/or information.

Positioning and navigation system 160 may determine information associated with objects, eg, end devices 130, vehicles 110, and the like. In some embodiments, the positioning and navigation system 160 may be a Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Compass Navigation System (COMPASS), Beidou Navigation Satellite System, Galileo Positioning System, Quasi-Zenith Satellite System ( QZSS) etc. The information may include the object's position, height, velocity or acceleration, current time, and the like. Positioning and navigation system 160 may include one or more satellites, such as satellite 160-1, satellite 160-2, and satellite 160-3. The satellites 160-1 to 160-3 may independently or collectively determine the above information. The positioning and navigation system 160 may transmit the above-mentioned information to the network 150 , the terminal device 130 or the vehicle 110 via a wireless connection.

Those of ordinary skill in the art will understand that when an element (or component) of data processing system 100 executes, the element may execute through electrical and/or electromagnetic signals. For example, when terminal device 130 sends a request to server 120, the processor of terminal device 130 may generate an electrical signal that encodes the request. The processor of the terminal device 130 may then transmit the electrical signal to the output port. If the terminal device 130 communicates with the server 120 via a wired network, the output port may be physically connected to a cable, which may also transmit electrical signals to the input port of the server 120 . If the terminal device 130 communicates with the server 120 via a wireless network, the output port of the terminal device 130 may be one or more antennas that convert electrical signals into electromagnetic signals. Within an electronic device such as terminal device 130 and/or server 120, when its processor processes instructions, issues instructions, and/or performs actions, the instructions and/or actions are performed through electrical signals. For example, when a processor retrieves or saves data from a storage medium (eg, storage device 140), it can send electrical signals to a read/write device of the storage medium, which can read or write in the storage medium into structured data. The structured data can be sent to the processor in the form of electrical signals over the bus of the electronic device. Here, an electrical signal may refer to one electrical signal, a series of electrical signals, and/or a plurality of discrete electrical signals.

Figure 2 is a block diagram of an exemplary processing device shown in accordance with some embodiments of the present application. In some embodiments, processing device 122 may be used for data storage. As shown in FIG. 2 , the processing device 122 may include an original data set acquisition module 210 , a target data set establishment module 220 , an index establishment module 230 and a storage module 240 .

The raw data set obtaining module 210 can be used to obtain a raw data set, the raw data set includes a plurality of data elements, and each data element has type information marking the type of the data element. In some embodiments, a data set may refer to a data set that includes a plurality of data elements. In some embodiments, the dataset may be a file, and the data elements of the file are messages. In some embodiments, different data sets and/or data elements in the same data set may have respective identification information.

In some embodiments, taking drive test data as an example, the raw data set obtaining module 210 may obtain the raw data set (ie, drive test data) from a test vehicle (eg, vehicle 110 ) via the network 150 . Specifically, the drive test data may be a message with a temporal nature. The original data set acquisition module 210 may organize the messages collected by a test vehicle during a test trip into a file (for example, a bag file) for storage to obtain an original data set. Further, the original data set acquisition module 210 may also use identification information related to the test vehicle and the itinerary as the identification information of the file. For example, the identification information of the file may be set according to the id of the test vehicle and the id of the test itinerary. In some embodiments, the original data set obtaining module 210 may further use the time information of the message as the identification information of the message in the file. For example, the identification information of the message in the file may be set according to the timestamp of the message.

In some embodiments, each data element in the original data set has type information that identifies the type of the data element. In some embodiments, the types may include one or more of an image class, a location class, a sensor class, a packet class, and a Controller Area Network Bus (CAN Bus) class.

The target data set establishment module 220 may be configured to establish different target data sets according to the type information of the data elements in the original data set. Each type of data element can correspond to a target data set. For example, the number of types of data elements in the original data set is N, and N different target data sets can be established, and the N different target data sets correspond to different types of data elements, where N is Integer greater than or equal to 2. In some embodiments, the target data set establishment module 220 may identify the type of each data element in the original data set and determine the number of different types. Further, the target data set establishment module 220 may establish different target data sets corresponding to different types of data elements. For example, the type of data element can be represented by the type of the device that acquires the drive test data. The devices may include cameras, radars, inertial measurement units (IMUs), and the like. The raw data may include camera data, radar data, IMU data, and the like. The different target datasets may include camera-type target datasets, radar-type target datasets, and IMU-type target datasets. For another example, the type of a data element can be expressed by a data type. The data types may include audio data, image data, text data, and the like. The different target data sets may include audio target data sets, image target data sets, text target data sets, and the like.

In some embodiments, the target data set establishment module 220 may further set the identification information of the target data set according to the corresponding type of the target data set, so as to identify different types of target data sets. Since the identification information and type information of the target data set are corresponding, in some embodiments, the identification information may include type information common to all data elements in the same target data set.

The index building module 230 may build an index of the original data set to provide an indexing function for data elements in the original data set. In some embodiments, the index building module 230 may determine the meta-service information of the original data set, which may also be referred to as meta-information. The meta-service information may be used to describe the structure, semantics, purpose, usage, etc. of the original data set or data elements in the original data set. In some embodiments, the meta-service information may also be referred to as index information or include index information for determining the storage location of the original data set or data elements in the original data set in the storage device. In some embodiments, the meta-service information may at least include meta-identification information (eg, timestamp) and storage location information (eg, offset) corresponding to each data element in the target data set. Wherein, the meta identification information may refer to identification information (eg, timestamp) of the corresponding data element. In some embodiments, the meta-service information may further include identification information of the target data set, where the identification information of the target data set corresponds to the type. In some embodiments, the meta-service information of the target data set may further include set identification information of the original data set corresponding to each data element in the target data set (eg, the id of the test vehicle and/or the id of the test trip, etc.).

In some embodiments, indexing module 230 or storage module 240 may store the meta-service information in a storage device, eg, storage module 240, storage device 140, or other storage device, processing device 122 (eg, calling module 320). ) can access the storage device based on the user's data invocation request, and further locate the data element corresponding to the user's data invocation request based on the meta-service information and the identification information corresponding to the target data set (for example, an image target data set) , that is, determine the storage location of the data element in the storage device. In some embodiments, the index building module 230 may build meta-service information for each target dataset separately. The meta-service information of each target data set is stored in a storage device in the form of a list, for example, the storage device 140 or other storage devices.

The storage module 240 may be configured to store the data elements corresponding to the target data set in the corresponding target data set based on the type information of the data elements in the original data set and the target data set. In some embodiments, the storage module 240 may determine the type of each data element in the original data set and store the data element in the corresponding target data set. For example, if the data element is an image-like data element, the processing device 122 may store the data element in an image-like object dataset. Further, the storage module 240 may also store the target dataset (eg, an image-type target dataset) and the data elements stored therein in a storage device, eg, the storage device 140 (eg, a distributed file system). In some embodiments, the storage module 240 may store each data element in the same original data set in a physically contiguous memory space, or may store it in a physically non-contiguous memory space and the data elements stored in the non-contiguously pass through between the data elements. pointer to link. For example only, in some embodiments, the storage module 240 may store the data elements in the target dataset in a distributed file system (HDFS), which may be physical storage.

In some embodiments, the storage module 240 may also store meta-service information corresponding to the target dataset (eg, an image-type target dataset) in a storage device, eg, the storage device 140 (eg, a distributed file system) )middle. The meta-service information may point to the target data set through a pointer. The user can locate the target dataset through the meta-service information. In some embodiments, the storage module 240 may store the meta-service information corresponding to the target dataset in the form of a list. In some embodiments, the storage device used to store the target dataset and the data elements therein may be the same or different from the storage device used to store the meta-service information.

It should be noted that the above description of the processing device 122 and its modules is only for convenience of description, and cannot limit the present application to the scope of the illustrated embodiments. It can be understood that for those skilled in the art, after understanding the principle of the system, various modules may be combined arbitrarily, or a subsystem may be formed to connect with other modules without departing from the principle. For example, the original data set acquisition module 210, the target data set establishment module 220, the index establishment module 230, and the storage module 240 disclosed in FIG. 2 may be different modules in a system, or may be one module to implement the above two or both functions of more than one module. For example, the storage module 240 and the index building module 230 may be two independent modules, or one module may have the functions of data storage, index building and caching at the same time. Such deformations are all within the protection scope of the present application.

FIG. 3 is a block diagram of another exemplary processing device shown in accordance with some embodiments of the present application. In some embodiments, processing device 122 may be used to invoke data. As shown in FIG. 3 , the processing device 122 may include a user request obtaining module 310 and a calling module 320 .

The user request acquisition module 310 may be configured to acquire a user's data invocation request, where the data invocation request at least includes the type of the data to be invoked. In some embodiments, the data call request may be input by the user through a mobile device (eg, an input/output interface of the terminal device 130 ) or a computing device. For example, the input/output interface of the terminal device 130 may include an input device, such as a keyboard, a mouse, a touch screen, a microphone, a trackball, etc., or any combination thereof, and the user may use the input device to input the data calling request. In some embodiments, the user request acquisition module 310 may acquire the data call request (eg, via the network 150). In some embodiments, the data includes drive test data. The data invocation request may also include information such as the time range information of the drive test data collection, the id of the test vehicle and/or the id of the test trip.

The calling module 320 may be configured to obtain the data elements in the target data set of the corresponding type based on the data calling request, and obtain the data to be called. In some embodiments, the calling module 320 may include an index information obtaining unit 322 , a segment storage unit 323 , and a data element obtaining unit 324 .

The index information obtaining unit 322 may be configured to obtain index information corresponding to the data calling request based on the data calling request. The data element obtaining unit 324 may be configured to obtain the data element based on the index information. In some embodiments, data processing system 100 may provide an indexing mechanism. For example, the indexing module 230 may determine meta-service information for the target dataset, which may be stored in a storage device. The index information obtaining unit 322 may obtain the user's index request information from the data calling request. Further, the data element obtaining unit 324 may match the index request information in the data invocation request with the meta-service information stored in the storage device, thereby determining index information (or index information matching the index request information in the data invocation request). meta-service information), to obtain data elements based on the index information or the storage location pointed to by the meta-service information.

In some embodiments, the data invocation request at least includes the type of the data to be invoked. In some embodiments, the type described by the data to be called may include one or more than one. Since the identification information and type information of the target data set are corresponding, the index information obtaining unit 322 can access the meta-service information based on one or more types (that is, user index request information) selected by the user included in the data call request, Index information in the meta-service information that matches the one or more types is determined. Further, the data element obtaining unit 324 may determine the storage location of the corresponding target data set based on the index information, so as to call the data element in the corresponding target data set.

In some embodiments, the data invocation request may further include more filter conditions related to the data to be invoked. For example, in some embodiments, the index information may include at least one-to-one corresponding meta identification information (eg, timestamp) and storage location information of each data element in the target data set. Correspondingly, the data invocation request may further include meta-qualification conditions (eg, time range) related to the meta-identification information for the data to be invoked. The index information obtaining unit 322 may access the meta-service information based on the data call request, and determine the index information in the meta-service information that corresponds to the one or more types and satisfies the meta-qualifying condition. Further, the data element obtaining unit 324 may obtain the data element according to the storage location corresponding to the index information.

In some embodiments, the index information may further include set identification information of the original data set corresponding to each data element in the target data set, wherein the set identification information refers to the identification information of the original data set. Correspondingly, the data invocation request may further include a set qualification related to the set identification information for the data to be invoked. The index information obtaining unit 322 may access meta-service information based on the data call request, and determine index information in the meta-service information that corresponds to the one or more types and satisfies the meta-qualifying condition and/or the set-qualifying condition. Further, the data element obtaining unit 324 may obtain the data element according to the storage location corresponding to the index information.

In some embodiments, the data may include drive test data. The user's data invocation request may include index request information such as the type of the drive test data collection device, the time range information of the drive test data collection, the id of the test vehicle and/or the id of the test trip. The index request information determined by the data calling request may include at least the id of the test vehicle and/or the id of the test trip, the type of the data to be called, the time range, the data time length, and the like.

In some embodiments, the segment storage unit 323 may be configured to obtain the target data set stored in the target data set from a storage device (eg, a distributed file system) in which the target data set is stored based on the meta-service information matched with the user's data invocation request. data element. Further, the segment storage unit 323 may further store the target data set according to the preset time information (eg, timestamp) based on the time information (for example, time stamp) identified by the data element, that is, the time information corresponding to the target data set and the time information in the user data call request. The time interval is divided into multiple target data subsets, each target data subset corresponds to a time interval, and the data elements obtained at each time interval (for example, every 10s) are stored in each target data subset (also called physical data). data files). For example, when the time range length in the user data call request is smaller than the time range length of the target data set, the target data set is divided into multiple target data subsets at preset time intervals. For example, if the time length of the data element corresponding to each target data set is 100 seconds, and the time length in the user data call request is 20 seconds, the segment storage unit 323 may divide the target data set into 10 target data subsets, Each target data subset corresponds to 10 seconds of data elements. As described herein, the time range here may refer to the time range in which the data elements are collected.

In some embodiments, the segment storage unit 323 may store each target data subset and the data elements stored therein in the memory of the processing device 122 by means of physical storage. After the user completes the invocation of some data elements in the target data set, the target data subset and the data elements stored therein can be erased. Further, the data obtaining unit 324 may obtain, from the storage device, data elements in the target data subsets matching the time information in each time interval based on the time information of the user data call request. For example, when the user's location is in the same city or country as the storage device (referred to as the first storage device, eg, a distributed file system) that stores the original data set, the processing device 122 (referred to as the local server) may distribute the data based on the above method To the user's terminal (also referred to as the user terminal, eg, terminal device 130).

In some embodiments, the segment storage unit 323 may determine the target data set pointed to by the meta-service information based on the meta-service information matching the user's data invocation request. And based on the time information (eg, time range) corresponding to the target data set and the time information of the user data invocation request, multiple logical files corresponding to the target data set are established. For example, when the location of the client is not in the same area (eg, city or country) as the storage device (referred to as the first storage device, such as a distributed file system) storing the original data set, the processing device 122 and the first storage device The storage devices are in the same region (eg, city or country), and the processing device 122 can create multiple logical files corresponding to the target dataset. The processing device 122 may further send the target data set and its stored data elements to a second storage device located in the same region (eg, city or country) as the client. The server where the second storage device is located may divide the received target data set (physical data file) into multiple target data subsets at preset time intervals, and store the data elements in the corresponding target data subsets respectively. The plurality of logical files point to the target data subset in the second storage device by way of pointers. The data acquisition unit 324 can determine the logical file matching the user invocation request by matching the time information in the user data invocation request with the time information in each logical file, and based on the second storage device pointed to by the logical file matched by the user. the target data subset, instructing the server of the second storage device to send the data elements in the matched target data subset to the client. Further, the server of the second storage device may combine the data elements in the multiple target data subsets and send them to the client.

The data elements in the target data subset are not stored in the logical file, and information about the data elements (eg, part of the meta-service information) may be stored. Logical files can point to physical data files (ie, target data subsets) by way of pointers. For example, the target data set can be divided into multiple target data subsets according to preset time intervals, each target data subset corresponds to a time interval, and each target data subset can create a logic file. Each logical file includes meta-service information of data elements stored in each target data subset.

The user may locate the target sub-data set in the second storage device through the time information in the user call request through the meta-service information in the logical file. In some embodiments, the logical file and the physically stored target data subset may be linked by a pointer. In some embodiments, the second storage device may store each data element in the same target sub-data set in a physically contiguous memory space, or may store it in a physically non-contiguous memory space. For example only, in some embodiments, the second storage device may store the data elements in the target sub-dataset in a distributed file system (HDFS), which may be physical storage. Correspondingly, the user can call the data actually corresponding to the specified time period (that is, the physically stored target data subset) from the second storage device located in the same area as the user terminal by accessing the logical file in the first storage device. , so as to realize the function of quickly calling part of the data. In some embodiments, the segment storage unit 323 may set the time interval according to the minimum value of the time period specified by the user for the data to be called, so as to ensure that several target data subsets called according to the time period are related to the time interval as much as possible. The actual corresponding data of the segment are consistent with each other. In some embodiments, the segment storage unit 323 may directly set the minimum value of the time period specified by the user for the data to be called as the time interval.

In some embodiments, the segment storage unit 323 may determine and acquire the target data set from the storage device storing the original data set based on the time information (eg, timestamp) identified by the data element, and store the target data set according to a preset value. The time interval is divided into multiple target data subsets, and each target data subset is stored separately. Further, when the data call request also includes the time range corresponding to the data to be called, and the time range corresponding to the data to be called is smaller than the time range corresponding to the target data set, the processing device 122 does not need to send the data in the entire target data set. For the client, it is only necessary to send the data elements corresponding to the time range information in the user's call data request (that is, the data elements in the target data subset) to the client, so as to realize the function of quickly calling part of the data and improve the efficiency of data calling.

It should be noted that the above description of the processing device 122 and its modules is only for convenience of description, and cannot limit the present application to the scope of the illustrated embodiments. It can be understood that for those skilled in the art, after understanding the principle of the system, various modules may be combined arbitrarily, or a subsystem may be formed to connect with other modules without departing from the principle. For example, the user request acquisition module 310 and the invocation module 320 disclosed in FIG. 3 may be different modules in a system, or may be a module that implements the functions of the above-mentioned two or more modules. For example, the user request obtaining module 310 and the calling module 320 may be two modules, or one module may have the functions of obtaining user requests and calling data at the same time. Such deformations are all within the protection scope of the present application.

It should be understood that the system and its modules shown in Figures 2 and 3 may be implemented in various ways. For example, in some embodiments, the system and its modules may be implemented in hardware, software, or a combination of software and hardware. Wherein, the hardware part can be realized by using dedicated logic; the software part can be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or specially designed hardware. Those skilled in the art will appreciate that the methods and systems described above may be implemented using computer-executable instructions and/or embodied in processor control code, for example on a carrier medium such as a disk, CD or DVD-ROM, such as a read-only memory (firmware) ) or a data carrier such as an optical or electronic signal carrier. The system and its modules of the present application can not only be implemented by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc. , can also be implemented by, for example, software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software (eg, firmware).

FIG. 4 is an exemplary flowchart of a data storage method according to some embodiments of the present application. As shown in Figure 4, the data storage method may include:

Step 410: Obtain an original data set, where the original data set includes a plurality of data elements, and each data element has type information indicating the type of the data element. In some embodiments, step 410 may be performed by processing device 122 (eg, raw data set acquisition module 210).

In some embodiments, a data set may refer to a data set that includes a plurality of data elements. For example, when the amount of data is relatively large, it is desirable to organize the data into larger units and store them separately. In this case, a unit is a dataset. In some embodiments, the dataset may be a file, and the data elements of the file are messages. Further, the file may be in a bag format, hereinafter referred to as a "bag file". In some embodiments, the data elements in the data set may be associated with each other. In some embodiments, different data sets and/or data elements in the same data set may have respective identification information. Taking the drive test data as an example, the drive test data collected by the test vehicle may be a message with a temporal nature (for example, the message may have a time stamp). The processing device 122 may organize the messages collected by a test vehicle during a test trip into a file (eg, a bag file) for storage to obtain an original data set. Further, the processing device 122 may also use identification information related to the test vehicle and the itinerary as the identification information of the file. For example, the identification information of the file may be set according to the id of the test vehicle and the id of the test trip. In some embodiments, the processing device 122 may also use the time information of the message as the identification information of the message in the file. For example, the identification information of the message in the file may be set according to the timestamp of the message.

In some embodiments, the data element can mark the type to which it belongs by carrying type information. That is, a storage unit with a size capable of accommodating the data element and its type information can be allocated to the data element, and the data element and its type information can be organized together according to preset rules for storage. In some embodiments, the data element and the corresponding type information may be connected by a preset connection symbol, one side of the connection symbol is the data element, and the other side of the connection symbol is the type information of the data element. In some embodiments, a storage unit for storing a data element and its type information may be divided into at least two partitions, including a first partition for storing the data element itself and a first partition for storing the type information of the data element Second division.

Step 420: Establish different target data sets according to the type information of the data elements in the original data set. Each type of data element can correspond to a target data set. For example, the number of types of data elements in the original data set is N, and N different target data sets can be established, and the N different target data sets correspond to different types of data elements, where N is Integer greater than or equal to 2. In some embodiments, step 420 may be performed by processing device 122 (eg, target dataset establishment module 220).

In some embodiments, processing device 122 may identify the type of each data element in the original data set and determine the number of different types. For example, taking drive test data as an example, the original data set may include three types of data elements: image type, position type, and speed type. The processing device 122 may determine that the number of types of data elements in the original data set is three. Further, the processing device 122 may establish three different target data sets corresponding to different types of data elements. For example, the three different target datasets may be an image class target dataset, a location class target dataset, and a speed class target dataset. For another example, the types of data elements may be divided according to the type of the device that acquires the drive test data. The devices may include cameras, radars, inertial measurement units (IMUs), and the like. The raw data may include camera data, radar data, IMU data, and the like. The processing device 122 may establish different target data sets corresponding to different types of data elements. For example, the different target datasets may be camera-type target datasets, radar-type target datasets, and IMU-type target datasets.

In some embodiments, the processing device 122 may further set the identification information of the target data set according to the type corresponding to the target data set, so as to identify different types of target data sets. Since the identification information and type information of the target data set are corresponding, in some embodiments, the identification information may include type information common to all data elements in the same target data set.

For example only, in some embodiments, the processing device 122 may store a target dataset in one file and store identification information corresponding to the target dataset in another file, the target dataset and The corresponding identification information can be connected by preset connection symbols. The preset side of the connection symbol is the target data set, and the other side of the connection symbol is the identification information of the target data set.

Step 430: Based on the type information of the data elements in the original data set and the target data set, store the data elements corresponding to the target data set in the corresponding target data set. In some embodiments, step 430 may be performed by processing device 122 (eg, storage module 240).

In some embodiments, processing device 122 may determine the type of each data element in the original dataset and store the data element in the corresponding target dataset. For example, if the data element is an image-like data element, the processing device 122 may store the data element in an image-like object dataset. For example, FIG. 5 is a schematic diagram of storing different types of data elements in the original data set in the corresponding target data set according to some embodiments of the present application. As shown in FIG. 5 , the data element types of an original data set include three types A, B, and C, and three target data sets are established corresponding to these three types respectively. Further, the data elements A1, A2 and A3 belonging to type A are stored in the target data set corresponding to type A, the data elements B1 and B2 belonging to type B are stored in the target data set corresponding to type B, and the data elements belonging to type C are stored in the target data set corresponding to type B. Data elements C1, C2, C3, and C4 are stored in the target dataset corresponding to type C.

Further, the processing device 122 may also store the identification information corresponding to the target data set in the storage device as meta-service information. In the storage device, the identification information may point to the image class target dataset through a pointer. In some embodiments, the user may locate the image-like target dataset based on the meta-service information. For example, a user may input a query request related to the image-based target data set through the input/output interface of the terminal device 130, and the processing device 122 may access the storage device based on the query request, thereby determining the image-based target data set location.

In some embodiments, each data element in the same original data set may be stored in a physically contiguous memory space, or may be stored in a physically non-contiguous memory space, and the data elements stored in a non-contiguous manner are linked by pointers. In some embodiments, the processing device 122 (eg, the index building module 230 ) may also build index information of the target data set to provide an indexing function for data elements in the target data set. In some embodiments, the index information may include at least one-to-one corresponding meta identification information and storage location information of each data element in the target data set. Wherein, the meta identification information may refer to the identification information of the corresponding data element. Based on this, once the meta identification information of the data element is determined, the storage location information of the data element corresponding to the meta identification information of the data element can be determined, so that the data element can be called according to the determined storage location information. In some embodiments, the index information may further include identification information of the target dataset. Based on this, the processing device 122 may determine the location of the target dataset to be invoked based on the identification information of the dataset in the user's data invocation request. In some embodiments, the index information may be stored in a storage device, and the processing device 122 may access the storage device based on the user's data call request, and further locate the data call with the user based on the index information in the storage device Request the corresponding data element. In some embodiments, processing device 122 may establish index information for each target data set separately.

In some embodiments, the meta identification information may include time information of the corresponding data element. Further, in some embodiments, the time information may include a time stamp. Timestamps can be used to uniquely identify when a piece of data was generated (eg, when the data element was collected). In some embodiments, the storage location information may include an offset. The offset may refer to the distance between the actual address of the storage unit (eg, the address of the data element) and the segment address of the segment in which it is located (eg, the target data set). For the specific implementation of the index information, reference may be made to FIG. 6 and related descriptions. FIG. 6 is a schematic diagram of index information corresponding to a target dataset according to some embodiments of the present application. As shown in FIG. 6 , the type file represents the target data set, and the type index file represents the index information of the target data set. In the type index file, Timestamp represents the timestamp and Offset represents the offset. Each message (indicated by Msg) in the type index file corresponding to the type file points to (links to) each message in the type file, and includes the time stamp and storage location of the pointed message. Based on the build type index file, the corresponding message can be determined by the time stamp. Similarly, it can be understood that by establishing index information, corresponding data elements can be determined by time information. For example, when the user wants to call data of a specific time period, the system can obtain the time period specified by the user and query the index information containing the time information belonging to the time period, and according to the time information belonging to the time period in the queried index information The corresponding storage location information determines the location of the data element belonging to the time period, so that the corresponding part of the data element is called according to the time period specified by the user.

In some embodiments, the data elements in the target data set may be stored contiguously in chronological order. Based on this, for the time period specified by the user for the data to be called, the system can determine the start time and end time of the time period, and determine the corresponding start time according to the start time and end time and the established index information. The data element (called "starting data element") and the data element corresponding to the termination time (called "terminating data element"), and then call all the data elements from the starting data element to the ending data element (that is, all data elements belonging to the time period). Specifically, referring to Fig. 6, the messages in the type file are stored continuously in the order of their respective timestamps. Start offset and end offset corresponding to the start timestamp and end timestamp, and then determine the corresponding start message and end message in the type file according to the start offset and end offset, and call the start and end messages from the All messages from the Start message to the Termination message (eg, Msg3 to Msg5 in Figure 6). In this way, all data elements belonging to a specified time period can be called by locating the start and end data elements, which is improved compared to calling all data elements belonging to a specified time period by locating each data element belonging to a specified time period. data call efficiency.

It should be noted that, in some embodiments, each data element in the original data set may also be continuously stored in time sequence. In this way, when the original data set is divided into multiple target data sets according to the type of data elements, the data elements of the same type determined from the original data set can be spliced together in the original order in the original data set, so as to obtain The target dataset in which each data element is stored consecutively in chronological order.

In some embodiments, the index information of the target data set may further include set identification information of the original data set corresponding to each data element in the target data set. Referring to the foregoing, the set identification information may include the id of the test vehicle and/or the id of the test trip. Based on this, according to the index conditions set by the user for the set identification information, only the data elements in the target dataset that meet the conditions can be called. For example, in some embodiments, the index condition may include a range of test vehicle ids, a range of test trip ids, the ids of a specific one or more test vehicles/test trips, etc., or any combination thereof.

It should be noted that the above description of the data storage method 400 is only for the convenience of description, and does not limit the present application to the scope of the illustrated embodiments. It can be understood that for those skilled in the art, after understanding the principle of the method, various steps may be combined arbitrarily without departing from the principle, or any steps may be added or deleted.

By dividing the original data set into multiple target data sets corresponding to different types for storage, the user can directly obtain data from the corresponding target data set according to the type of the data to be called when calling data, rather than extracting some of the types based on the original data set The calling method is direct and the amount of data called is small, so it can efficiently call part of the data that meets the specific needs of users from a huge amount of data. With the data storage method provided by the embodiment of the present application, data elements of the same type in the original data set can be stored in a target data set, and the system only needs to find and access the target data set belonging to the user-specified type, and then the user-specified type can be called from it. The data. In addition, each data element in the target data set can be continuously stored in chronological order, and the user can further obtain target data of a specified type within a specified time period. Further, the target data set can be divided into multiple target data subsets at preset time intervals and cached respectively, and the user can obtain only the data elements corresponding to some target data subsets in the multiple target data subsets based on the data call request. In order to realize the function of quickly calling part of the data. Compared with calling data based on the original data set, the data storage method provided by the embodiment of the present application makes the data calling process simpler and the amount of data access smaller, and can better improve the efficiency of data calling.

FIG. 7 is an exemplary flowchart of a data calling method according to some embodiments of the present application. As shown in Figure 7, the data calling method may include:

Step 710: Acquire a data invocation request sent by the client, where the data invocation request at least includes the type of the data to be invoked. In some embodiments, step 710 may be performed by processing device 122 (eg, user request acquisition module 310).

In some embodiments, the data call request may be input by the user through a mobile device (eg, an input/output interface of the terminal device 130 ) or a computing device. For example, the input/output interface of the terminal device 130 may include an input device, such as a keyboard, a mouse, a touch screen, a microphone, a trackball, etc., or any combination thereof, and the user may use the input device to input the data calling request. In some embodiments, the data invocation request may be further sent (eg, via network 150 ) to processing device 122 and/or other components of data processing system 100 . For example only, the mobile device or computing device may provide a data query interface, which may support the user to input filter conditions related to the data to be called. The mobile device or computing device generates a corresponding data call request after acquiring the filter condition input by the user, and sends the data call request to the processing device 122 and/or other components of the data processing system 100 . In some embodiments, the data includes drive test data. The data invocation request may also include information such as the time range information of the drive test data collection, the id of the test vehicle and/or the id of the test trip.

Step 720: Obtain partial data in the original data based on the data calling request to obtain the data to be called, where the partial data includes data elements in the target data set corresponding to the type to which the data to be called belongs. In some embodiments, step 720 may be performed by processing device 122 (eg, calling module 320).

In some embodiments, data processing system 100 may provide an indexing mechanism. For example, as described in FIG. 4 , the processing device 122 may determine meta-service information of the target dataset to provide an indexing function for data elements in the target dataset. The meta-service information may be stored in a storage device. The processing device 122 (for example, the index information obtaining unit 322) may obtain the user's index request information from the data call request. Further, the processing device 122 may compare the index request information in the data call request with the data stored in the storage device. Match the meta-service information in the data call request to determine the index information (or meta-service information) that matches the index request information in the data call request, and obtain the data element based on the storage location pointed to by the index information or the meta-service information.

In some embodiments, the data invocation request at least includes the type of the data to be invoked. In some embodiments, the type described by the data to be called may include one or more than one. For example, the data query interface provided by the mobile device or computing device can display multiple candidate types. After the user selects one or more types to which the data to be called belongs, the mobile device or computing device generates one or more types including the one or more types selected by the user. type of data call request and send the data call request to data processing system 100 . In some embodiments, the index information may include at least identification information of the target dataset. Since the identification information and type information of the target data set are corresponding, the processing device 122 can access the meta-service information based on one or more types selected by the user (that is, the user index request information) included in the data call request, and determine the meta-service information. Index information corresponding to the one or more types in the service information. Further, the processing device 122 may determine the storage location of the corresponding target data set based on the index information, so as to call the data elements in the corresponding target data set.

In some embodiments, the data invocation request may further include more filter conditions related to the data to be invoked. It should be noted that, when the data invocation request includes multiple types selected by the user, the processing device 122 may acquire the data to be invoked that meets the multiple filtering conditions in various ways. For example, in some embodiments, the processing device 122 may first determine multiple target data sets corresponding to multiple types selected by the user one-to-one, and then select the target data sets of each type according to other filtering conditions in the data call request. The data set filters out the data elements that meet the conditions, so as to obtain the data to be called that satisfies multiple filter conditions. For another example, in some embodiments, the processing device 122 may first filter out data elements that meet other filtering conditions from all types of target data sets, and then filter out data elements belonging to multiple types selected by the user from the filtered data elements. data elements, so as to obtain the data to be called that satisfies multiple filter conditions. For another example, the processing device 122 may access the meta-service information based on the data invocation request, and determine the storage location of the data to be invoked that satisfies multiple screening conditions based on the index information in the meta-service information, so as to obtain the data to be invoked that satisfies the plurality of screening conditions. data.

In some embodiments, the index information may include at least one-to-one correspondence of meta identification information and storage location information of each data element in the target data set, wherein the meta identification information refers to identification information of a corresponding data element. Correspondingly, the data invocation request may further include meta-qualification conditions related to meta-identification information for the data to be invoked. The processing device 122 may access meta-service information based on the data call request, and determine index information in the meta-service information that corresponds to the one or more types and satisfies the meta-qualifying condition. Further, the processing device 122 may acquire the data element according to the storage location corresponding to the index information.

In some embodiments, data processing system 100 may provide an edge caching mechanism. For example, in some embodiments, processing device 122 (eg, segment storage unit 323 ) may retrieve data from a storage device (eg, a distributed file system) that stores the target dataset based on meta-service information that matches the user's data invocation request. to obtain the data elements stored in the target dataset. Further, the processing device 122 may divide the target data set according to preset time intervals based on the time information (eg, timestamp) identified by the data element, that is, the time information corresponding to the target data set and the time information in the user data call request. Create multiple target data subsets, each target data subset corresponds to a time interval, and store the data elements acquired at each time interval (for example, every 10s) to each target data subset (also called a physical data file) . For example, when the time range length in the user data call request is smaller than the time range length of the target data set, the target data set is divided into multiple target data subsets at preset time intervals. For example, if the time length of the data element corresponding to each target data set is 100 seconds, and the time length in the user data call request is 20 seconds, the processing device 122 may divide the target data set into 10 target data subsets, each The target data subset corresponds to 10 seconds of data elements. As described herein, the time range here may refer to the time range in which the data elements are collected. In some embodiments, the time interval may be set according to the minimum value of the time period specified by the user for the data to be called, so as to ensure that several target data subsets called according to the time period are actually corresponding to the time period. match. In some embodiments, the minimum value of the time period specified by the user for the data to be called may be directly set as the time interval.

In some embodiments, the processing device 122 may store each target data subset and the data elements stored therein in the memory of the processing device 122 by means of physical storage. After the user completes the invocation of some data elements in the target data set, the target data subset and the data elements stored therein can be erased. Further, the processing device 122 (eg, the data obtaining unit 324 ) may obtain, from the storage device, data elements in the target data subset matching the time information in each time interval based on the time information of the user data call request. For example, when the user's location is in the same city or country as the storage device (referred to as the first storage device, eg, a distributed file system) that stores the original data set, the processing device 122 (referred to as the local server) may distribute the data based on the above method To the user's terminal (also referred to as the user terminal, eg, terminal device 130).

In some embodiments, the processing device 122 may determine the target dataset to which the meta-service information points based on the meta-service information that matches the user's data invocation request. And based on the time information (eg, time range) corresponding to the target data set and the time information of the user data invocation request, multiple logical files corresponding to the target data set are established. For example, when the location of the client is not in the same area (eg, city or country) as the storage device (referred to as the first storage device, such as a distributed file system) storing the original data set, the processing device 122 and the first storage device The storage devices are in the same region (eg, city or country), and the processing device 122 can create multiple logical files corresponding to the target dataset. The processing device 122 may further send the target data set and its stored data elements to a second storage device located in the same region (eg, city or country) as the client. The distance between the second storage device and the user terminal is smaller than the distance between the first storage device and the user terminal. The server where the second storage device is located may divide the received target data set (physical data file) into multiple target data subsets at preset time intervals, and store the data elements in the corresponding target data subsets respectively. The plurality of logical files point to the target data subset in the second storage device by way of pointers. The processing device 122 may determine the logical file matching the user invocation request by matching the time information in the user data invocation request with the time information in each logical file, and based on the data in the second storage device pointed to by the user-matched logical file. The target data subset, instructing the server of the second storage device to send the data elements in the matched target data subset to the client.

Step 730: Synchronize the data to be called to the storage device of the client. In some embodiments, step 730 may be performed by processing device 122 (eg, a synchronization module (not shown)). In some embodiments, the processing device 122 may further combine the data elements in the multiple target data subsets and send them to the storage device of the client, so as to realize the synchronization of the data to be called on the client.

According to the above method, when the data call request further includes the time range corresponding to the data to be called, and the time range corresponding to the data to be called is smaller than the time range corresponding to the target data set, the processing device 122 does not need to transfer the data in the entire target data set To send to the client, it is only necessary to send the data element corresponding to the time range information in the user's call data request (that is, the data element in the target data subset) to the client, so as to realize the function of quickly calling part of the data and improve the efficiency of data calling .

It should be noted that the above description of the data calling method 700 is only for convenience of description, and cannot limit the present application to the scope of the illustrated embodiments. It can be understood that for those skilled in the art, after understanding the principle of the method, various steps may be combined arbitrarily without departing from the principle, or any steps may be added or deleted. For example, step 720 may also include an intelligent recommendation process. Specifically, the processing device 122 may record the calling habits of the user, and recommend calling results to the user according to the calling habits. For another example, the processing device 122 may also predict the user's search behavior based on a machine learning algorithm.

FIG. 8 is a schematic diagram of a data calling process according to some embodiments of the present application. As shown in FIG. 8 , the data invocation request input by the user may include information related to the trip ID (ie, the id of the test trip), the time range and the type. The multiple target datasets may include A-type files, B-type files, and C-type files. According to the indexing mechanism provided by the data processing system 100, the processing device 122 can access the meta-service information based on the data call request. The meta-service information may include the type file index and type file information as shown in FIG. 8 , wherein the type file index may be index information related to the trip ID (that is, the id of the test trip), time range, etc., and the type file The information may be index information related to identification information of the target data set, where the identification information of the target data set corresponds to the type. The processing device 122 may determine, according to the type file information and the type file index, the target data set corresponding to the type, and the specific position (for example, the offset start point and the offset end point) of the data element to be called in each target data set, thereby Get the object to be called in each target dataset. Further, the processing device 122 may combine the data elements to be called to generate a data packet. The data packet can be transmitted to the client as a result of the data call.

FIG. 9 is a schematic diagram of a data calling scenario according to some embodiments of the present application. As shown in FIG. 9 , the data invocation scenario may include a client, a local server (and the data processing system 100 ), and a remote data center.

On the client side, the user may input a data call request through the client side, that is, the computing device (eg, the input/output interface of the terminal device 130).

In some embodiments, a local server (eg, data processing system 100) may include an upper file system (which may also be referred to as a logical file system) and an underlying file system. The upper file system (or logical file system) can be used to define the interface (ie access) between the local server and the client. For example, the upper file system may provide an indexing mechanism. For example, the index may be established by the processing device 122 based on the raw data set. The upper file system can also define information such as files and their attributes, operations allowed by files, and directories of files. Further, the processing device 122 can determine the index request information corresponding to the data invocation request according to the data invocation request through the upper-layer file system, and determine the meta-service information in the underlying file system corresponding to the invocation request based on the data index information and the file directory. , and determine the storage location of the data element that satisfies the data calling request based on the meta-service information (eg, storage location information), so as to obtain the data to be called through the underlying file system.

The underlying file system is used to map the upper file system to a physical storage device (eg, a hard disk in a local server) or a memory device. For example, the underlying file system may include meta-service information, including index information of the target data set (for example, the identification information of the target data set, the one-to-one meta-identification information and storage location information of each data element in the target data set, the original data set identification information for the set, etc.). The underlying file system can match the meta-service information based on the index request information determined in the upper-level file system, and determine the location where the data pointed to by the matching meta-service information is stored in the physical storage device, thereby obtaining data elements, and realizing the upper-level file system to Mapping between physical storage devices.

In some embodiments, in the underlying file system, the local server may provide an edge caching mechanism. For example, the local server may store the target data set in the remote data center to the storage device of the local server according to the method described in the process 400 . Just as an example, the local server may divide the target data set into multiple target data subsets at preset time intervals based on the time information (eg, timestamp) identified by the data element, and store them in the storage device respectively. The target data subset includes time information of the metadata, which may be the address at which the data element is stored on the storage device. Further, based on the time information of the user data call request, the local server may acquire, from the storage device, data elements in the target data subsets that match the time information in each time interval. For example, when the user's location is in the same area as the storage device that stores the original data set, the local server can distribute the data to the client based on the above method. As another example, when the location of the client is not in the same area as the storage device that stores the original data set (referred to as the first storage device), and the local server is in the same area as the first storage device, the local server can create a target Multiple logical files corresponding to the dataset. The local server may further send the target data set and the data elements stored therein to a second storage device, where the second storage device and the client are located in the same area. The server where the second storage device is located may divide the received target data set (physical data file) into multiple target data subsets at preset time intervals, and store the data elements in the corresponding target data subsets respectively. The plurality of logical files point to the target data subset in the second storage device by way of pointers. The local server can determine the logical file matching the user's invocation request by matching the time information in the user data invocation request with the time information in each logical file, and based on the target in the second storage device pointed to by the logical file matched by the user Data subset, instructing the server of the second storage device to send the data elements in the matched target data subset to the client. Further, the server of the second storage device may combine the data elements in the multiple target data subsets and send them to the client. According to the above method, when the data invocation request further includes the time range corresponding to the data to be invoked, the data actually corresponding to the time range can be invoked by accessing the meta-service information corresponding to the target data subset within the specified time range. Data (that is, a subset of the target data in physical storage), so that only the data actually corresponding to the time range can be synchronized, rather than the entire target data set, so as to realize the function of quickly calling part of the data and improve the efficiency of data calling. For example, as shown in FIG. 9 , a user can obtain data stored in a remote data center in a preset test station for further analysis and processing (eg, R&D program debugging, test simulation, problem data analysis, etc.).

FIG. 10 is a schematic diagram of data calling according to some embodiments of the present application. As shown in FIG. 10 , when the user needs to call data, a data call request can be input through the user terminal (step 1), and the data processing system 100 can obtain the meta-service information matching the call request from the meta-service module based on the data call request. (or index request information) (step 2). Further, the data processing system 100 can determine the target dataset to which the meta-service information points. As an example only, it is assumed that the data processing system 100 and the local storage device storing the original data set (referred to as the first storage distributed file system (HDFS)) are in the first area, and the client location and the second HDFS are in the second area (The first area and the second area are different areas) or the client is closer to the server of the second HDFS than the client is to the server of the first HDFS. For example, the first region may be located in the United States and the second region may be located in China. Correspondingly, the first HDFS may be a data center established in the United States, and the second HDFS may be a data center established in China (eg, Inner Mongolia (NMG) data center). After the data processing system 100 determines the target data set (for example, the camera data file) pointed to by the meta-service information, it can establish the target data set based on the time information (for example, the time range) corresponding to the target data set and the time information of the user data call request Corresponding multiple logical files (eg, logical camera data files). Further, the data processing system 100 may send the target dataset and its stored data elements to the second HDFS. The server where the second storage HDFS is located may divide the received target data set (physical data file) into multiple target data subsets at preset time intervals, and store the data elements in the corresponding target data subsets respectively. The plurality of logical files point to the target data subset in the second HDFS by way of pointers. The data processing system 100 can determine the logical file matching the user invocation request by matching the time information in the user data invocation request with the time information in each logical file, and determine the logical file in the second HDFS to which the logical file matched by the user points. Target data subset, instructing the server of the second HDFS to send the data elements in the matched target data subset to the client (step 3). Further, the server of the second HDFS may combine the data elements in the multiple target data subsets and send them to the client (step 4).

FIG. 11 is a schematic diagram of data storage and invocation according to some embodiments of the present application. As shown in FIG. 11, data processing system 100 includes a meta-service module that obtains data packets (as described elsewhere herein) for data packets (as described elsewhere herein). The meta-service module generates a packet processing task in response to the received packet (step 1). And send the data packet processing task and the data packet to the processing module. Specifically, the processing module may acquire a data packet (ie, original data set) processing task and process and store the data packet. For example, the processing module may process and store the original data packet (original data set) based on the process 400 described in FIG. 4 . The processing module may establish different target data sets according to the type information of the data elements in the original data packets. For example, if the number of types of data elements in the original data packet is N, N different target data sets may be established, and the N different target data sets correspond to different types of data elements. The processing module may also set the identification information of the target data set according to the type corresponding to the target data set, so as to identify different types of target data sets. Further, the processing module may determine the meta-service information (also referred to as index information or include index information) of the original data set, so as to determine the storage location of the original data set or data elements in the original data set in the storage device.

Based on the processing procedure, the processing module may generate a target data set and corresponding meta-service information. Further, the processing module can upload the target data set and its stored data elements (step 2) and store them in a local storage device or system (ie, the first distributed file system (HDFS)) (step 3), and store the meta-service information in the storage device associated with the meta-service module (step 4). As used herein, a local storage device or system refers to a storage device or system in the same region (eg, city or country) as data processing system 100 . In some embodiments, the first HDFS may synchronize the processed data packets to the second HDFS. The second HDFS is located in a different region (eg, a different city or country) from the first HDFS, so that the user terminal located in the region where the second HDFS is located can call data. For more description on data calling based on the second HDFS, reference may be made to FIG. 10 .

The meta-service information may include meta-identification information (for example, a timestamp) and storage location information (for example, an offset) that correspond to each data element in the target data set, the identification information of the target data set, and the corresponding data elements in the target data set. The set identification information of the original dataset, etc.

When the user needs to call the data, the user can obtain the download address or the access address of the data in the first HDFS (step 0). The user can invoke the request based on the user-side input data (step 5). The data invocation request may include information such as the type of the data to be invoked, meta-qualification conditions (eg, time range) related to the meta-identification information. For example, when the user wants to call the drive test data, the data call request may include index request information such as the type of the drive test data collection device, the time range information of the drive test data collection, the id of the test vehicle and/or the id of the test trip.

The data processing system 100 may obtain meta-service information (or index request information) matching the call request from the meta-service module based on the data call request (step 6), and call the distributed file system stored near the client according to the meta-service information (step 7) in the target data subset in HDFS. Wherein, calling data from a distributed file system near the client may refer to calling data from the first HDFS or the second HDFS (step 8). For example, when the location of the client is in the same area as the first HDFS where the original data set is stored, the data processing system 100 may acquire the data elements stored in the target data set from the first HDFS based on the metadata service information. Further, the data processing system 100 may, based on the time information (for example, timestamp) identified by the data element, that is, the time information corresponding to the target data set and the time information in the user data call request, store the target data set at preset time intervals. The target data subsets are divided into a plurality of target data subsets, and each target data subset and the data elements stored therein are stored in the memory of the data processing system 100 by means of physical storage. Based on the time information of the user data call request, the data processing system 100 may further acquire, from the storage device, data elements in the target data subsets that match the time information in each time interval.

For another example, the first HDFS may synchronize the processed data packets to the second HDFS. The second HDFS is located in a different region (eg, a different city or country) from the first HDFS, so that the user terminal located in the region where the second HDFS is located can call data. The distance between the second HDFS and the user terminal is smaller than the distance between the first HDFS and the user terminal. For more description on data calling based on the second HDFS, reference may be made to FIG. 10 . In some embodiments, the data processing system 100 may further combine data elements in multiple target data subsets obtained from a distributed file system (HDFS) near the client and send it to the client (step 9).

FIG. 12 is a schematic diagram of a user interaction interface according to some embodiments of the present application. As shown in FIG. 12 , the user interaction interface may include a time selection area 1410 , a type selection area 1420 , a data representation area 1430 , a download address area 1440 and a processing progress area 1450 .

In the time selection area 1410, the user can input the time range corresponding to the data to be recalled. For example, a user may input the time range through an input device (eg, keyboard, mouse, touch screen, microphone, trackball) associated with the user interface.

In the type selection area 1420, the user can input the type corresponding to the data to be called. For example, as shown in FIG. 12 , the user can input the type corresponding to the data to be called by checking the selection box corresponding to the type through an input device (eg, a mouse).

The data representation area 1430 may be used to display the data to be called corresponding to the time range and type input by the user. For example, as shown in FIG. 12 , the data to be called corresponding to the time range and type input by the user can be displayed in the data representation area 1430 in the form of a combination of timeline and data subset, so that the user can check or confirm whether the input information is not correct.

The download address area 1440 may be used to provide a download link corresponding to the data to be called. For example, the user can click on the download chain to trigger the data invocation process. The data to be called obtained during the data calling process can be combined to generate a file package, which is further downloaded on the user end.

The processing progress area 1450 may be used to display the progress of data processing. For example, as shown in FIG. 12 , the progress of data processing may include processing completed, unprocessed, and unable to process. The user can determine the progress of data processing (eg, data calling) through the processing progress area 1450 .

The possible beneficial effects of the embodiments of the present application include, but are not limited to: (1) Store data elements of the same type in the original data set in a target data set, and only need to find and access the target data set belonging to the type specified by the user. Call out the data of the type specified by the user; (2) the data elements in the target data set can be stored continuously in chronological order, and the user can further obtain the target data of the specified type within the specified time period; (3) the target data set can be preset The time interval is divided into multiple target data subsets, and the user can obtain only the data elements corresponding to some target data subsets in the multiple target data subsets based on the data calling request to realize the function of quickly calling partial data. Compared with calling data based on the original data set, the data storage method provided by the embodiment of the present application makes the data calling process simpler and the amount of data access smaller, and can better improve the efficiency of data calling. It should be noted that different embodiments may have different beneficial effects, and in different embodiments, the possible beneficial effects may be any one or a combination of the above, or any other possible beneficial effects.

The basic concept has been described above. Obviously, for those skilled in the art, the above detailed disclosure is only an example, and does not constitute a limitation to the present application. Although not explicitly described herein, various modifications, improvements and corrections to this application may occur to those skilled in the art. Such modifications, improvements, and corrections are suggested in this application, so such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this application.

Meanwhile, the present application uses specific words to describe the embodiments of the present application. Such as "one embodiment," "an embodiment," and/or "some embodiments" means a certain feature, structure, or characteristic associated with at least one embodiment of the present application. Therefore, it should be emphasized and noted that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various places in this specification are not necessarily referring to the same embodiment . Furthermore, certain features, structures or characteristics of the one or more embodiments of the present application may be combined as appropriate.

Furthermore, those skilled in the art will appreciate that aspects of this application may be illustrated and described in several patentable categories or situations, including any new and useful process, machine, product, or combination of matter, or combinations of them. of any new and useful improvements. Accordingly, various aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as a "data block", "module", "engine", "unit", "component" or "system". Furthermore, aspects of the present application may be embodied as a computer product comprising computer readable program code embodied in one or more computer readable media.

A computer storage medium may contain a propagated data signal with the computer program code embodied therein, for example, on baseband or as part of a carrier wave. The propagating signal may take a variety of manifestations, including electromagnetic, optical, etc., or a suitable combination. Computer storage media can be any computer-readable media other than computer-readable storage media that can communicate, propagate, or transmit a program for use by coupling to an instruction execution system, apparatus, or device. Program code on a computer storage medium may be transmitted over any suitable medium, including radio, cable, fiber optic cable, RF, or the like, or a combination of any of the foregoing.

The computer program coding required for the operation of the various parts of this application may be written in any one or more programming languages, including object-oriented programming languages such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python Etc., conventional procedural programming languages such as C language, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP, dynamic programming languages such as Python, Ruby and Groovy, or other programming languages. The program code may run entirely on the user's computer, or as a stand-alone software package on the user's computer, or partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network, such as a local area network (LAN) or wide area network (WAN), or to an external computer (eg, through the Internet), or in a cloud computing environment, or as a service Use eg software as a service (SaaS).

In addition, unless explicitly stated in the claims, the order of processing elements and sequences described in the present application, the use of numbers and letters, or the use of other names are not intended to limit the order of the procedures and methods of the present application. While the foregoing disclosure discusses by way of various examples some embodiments of the invention that are presently believed to be useful, it is to be understood that such details are for purposes of illustration only and that the appended claims are not limited to the disclosed embodiments, but rather The requirements are intended to cover all modifications and equivalent combinations falling within the spirit and scope of the embodiments of the present application. For example, although the system components described above may be implemented by hardware devices, they may also be implemented by software-only solutions, such as installing the described systems on existing servers or mobile devices.

Similarly, it should be noted that, in order to simplify the expressions disclosed in the present application and thus help the understanding of one or more embodiments of the invention, in the foregoing description of the embodiments of the present application, various features are sometimes combined into one embodiment, in the drawings or descriptions thereof. However, this method of disclosure does not imply that the subject matter of the application requires more features than those mentioned in the claims. Indeed, there are fewer features of an embodiment than all of the features of a single embodiment disclosed above.

Some examples use numbers to describe quantities of ingredients and attributes, it should be understood that such numbers used to describe the examples, in some examples, use the modifiers "about", "approximately" or "substantially" to retouch. Unless stated otherwise, "about", "approximately" or "substantially" means that a variation of ±20% is allowed for the stated number. Accordingly, in some embodiments, the numerical parameters set forth in the specification and claims are approximations that can vary depending upon the desired characteristics of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and use a general digit reservation method. Notwithstanding that the numerical fields and parameters used in some embodiments of the present application to confirm the breadth of their ranges are approximations, in particular embodiments such numerical values are set as precisely as practicable.

Each patent, patent application, patent application publication, and other material, such as article, book, specification, publication, document, etc., cited in this application is hereby incorporated by reference in its entirety. Application history documents that are inconsistent with or conflict with the content of this application are excluded, as are documents (currently or hereafter appended to this application) that limit the broadest scope of the claims of this application. It should be noted that, if there is any inconsistency or conflict between the descriptions, definitions and/or terms used in the attached materials of this application and the content of this application, the descriptions, definitions and/or terms used in this application shall prevail .

Finally, it should be understood that the embodiments described in the present application are only used to illustrate the principles of the embodiments of the present application. Other variations are also possible within the scope of this application. Accordingly, by way of example and not limitation, alternative configurations of embodiments of the present application may be considered consistent with the teachings of the present application. Accordingly, the embodiments of the present application are not limited to the embodiments expressly introduced and described in the present application.

Claims

A data storage method executed by a computing device, wherein the method comprises:

Obtaining an original data set, the original data set includes a plurality of data elements, and each data element has type information marking the type of the data element;

According to the type information of data elements in the original data set, the number N of different types is obtained, and N different target data sets are correspondingly established, and the N different target data sets correspond to different types of data elements; wherein , where N is an integer greater than or equal to 2; and

Based on the type information of the data elements in the original data set and each target data set, the data elements corresponding to the target data set are stored in the corresponding target data set, and the target data set is stored in the first storage device.
The method according to claim 1, wherein the data set is a file, and the data element of the file is a message.
The method of claim 1, wherein the types include one or more of an image type, a location type, a sensor type, a data packet type, and a controller area network bus type.
The method according to claim 1, wherein the method further comprises:

Establishing index information of the target data set, the index information at least includes meta-identification information and storage location information corresponding to each data element in the target data set; wherein, the meta-identification information refers to the identification information of the corresponding data element .
The method according to claim 4, wherein the data elements in the target data set are arranged in chronological order, and the element identification information includes time information of the corresponding data elements.
The method according to claim 4, wherein the index information further comprises set identification information of the original data set corresponding to each data element in the target data set; wherein, the set identification information refers to the identification information of the original data set .
The method according to claim 1, wherein the data in the original data set includes data generated or collected during the operation of the autonomous vehicle.
The method of claim 1, wherein:

Receive a data call request sent by the client, where the data call request at least includes the type of the data to be called;

Determine a corresponding type of target data set from the N different target data sets based on the data call request;

obtaining the data to be called based on the data elements in the determined target data set; and

Send the data to be called to the second storage device of the client.
The method according to claim 8, wherein the obtaining the data to be called based on the data elements in the determined target data set further comprises:

Obtain the target data set and the data elements stored therein from the first storage device;

dividing the target data set into a plurality of target data subsets at preset time intervals; and

Based on the data calling request, data elements corresponding to some target data subsets in the multiple target data subsets are acquired, and the data to be called includes the data elements corresponding to the partial target data subsets.
The method according to claim 8, wherein the obtaining the data to be called based on the data elements in the determined target data set further comprises:

Sending the determined target data set obtained in the first storage device and the data elements stored therein to a third storage device, where the distance between the first storage device and the user terminal is farther than the distance from the third storage device the client;

dividing the target data set into a plurality of target data subsets at preset time intervals and storing them in the third storage device;

creating a plurality of logical files, each logical file corresponding to one of the plurality of target data subsets, each logical file including index information corresponding to data elements in the target data subset; and

Based on the data calling request and the logic file, acquire, from the third storage device, data elements stored in a partial target data subset of the multiple target data subsets, where the data to be called includes the partial target the data element corresponding to the data subset.
A data storage system, characterized in that the system comprises:

an original data set acquisition module, used for acquiring an original data set, the original data set includes a plurality of data elements, each data element has type information marking the type of the data element;

The target data set establishment module is used to obtain the number N of different types according to the type information of the data elements in the original data set, and correspondingly establish N different target data sets, the N different target data sets are different from The data element of the type corresponds to; wherein, N is an integer greater than or equal to 2; and

The storage module is configured to store the data elements corresponding to the target data set in the corresponding target data set based on the type information of the data elements in the original data set and the target data set.
The system according to claim 11, wherein the data set is a file, and the data element of the file is a message.
The system of claim 11, wherein the types include one or more of an image type, a location type, a sensor type, a data packet type, and a controller area network bus type.
The system of claim 11, wherein the system further comprises:

An index information establishment module is used to establish index information of the target data set, and the index information at least includes meta identification information and storage location information corresponding to each data element in the target data set; wherein, the meta identification information is Refers to the identification information of the corresponding data element.
The system according to claim 14, wherein the data elements in the target data set are arranged in chronological order, and the element identification information includes time information of the corresponding data elements.
The system according to claim 14, wherein the index information further comprises set identification information of the original data set corresponding to each data element in the target data set; wherein, the set identification information refers to the identification information of the original data set .
A storage medium, characterized in that the storage medium is used to store computer instructions, and after a computer reads the computer instructions in the storage medium, the data storage method according to any one of claims 1 to 10 is executed. .
A data calling method executed by a computing device, wherein the data elements in the original data set are stored in the corresponding target data set according to the data storage method according to any one of claims 1 to 10, and the target data The data set is stored in the first storage device associated with the computer device, and the data calling method includes:

Obtain a data call request sent by the client, where the data call request at least includes the type of the data to be called;

Obtaining partial data in the target data set based on the data calling request to obtain the data to be called, the partial data including data elements in the target data set corresponding to the type to which the data to be called belongs; and

Send the data to be called to the second storage device of the client.
The method according to claim 18, wherein the target data set has corresponding index information, and the index information at least includes meta-identification information and storage location information corresponding to each data element in the target data set , wherein the meta-identification information refers to the identification information of the corresponding data element; the data invocation request also includes meta-qualification conditions related to the meta-identification information;

The acquiring data elements in the target data set of the corresponding type based on the data invocation request includes:

Obtain index information corresponding to the corresponding type and satisfying the meta-qualification condition based on the data call request; and

The data element is acquired based on the storage location in the acquired index information.
The method according to claim 19, wherein the data elements in the target data set are arranged in chronological order, the meta identification information includes time information of the corresponding data elements; the meta qualification includes the data to be called the corresponding time range.
The method according to claim 18, wherein the acquiring part of the data in the target data set based on the data invocation request further comprises:

dividing the target data set into a plurality of target data subsets at preset time intervals; and

Based on the data calling request, the data elements corresponding to some target data subsets in the target data subset are acquired, and the data to be called includes the data elements corresponding to the partial target data subsets.
The method according to claim 18, wherein the acquiring part of the data in the target data set based on the data invocation request further comprises:

Sending the target data set obtained in the first storage device and the data elements stored therein to a third storage device, where the distance from the first storage device to the user terminal is greater than the distance from the third storage device to the user terminal;

The target data set is divided into a plurality of target data subsets at preset time intervals and stored in the third storage device;

creating a plurality of logical files, each logical file corresponding to one of the plurality of target data subsets, the logical file including index information corresponding to data elements in the target data subset; and

Based on the data calling request and the logic file, data elements corresponding to some target data subsets in the plurality of target data subsets are acquired from the third storage device, and the data to be called includes the partial targets the data element corresponding to the data subset.
The method according to claim 19, wherein the index information further comprises set identification information of the original data set corresponding to each data element in the target data set, wherein the set identification information refers to the identification information of the original data set ; The data call request also includes set qualifications related to the set identification information;

The obtaining index information corresponding to the corresponding type and satisfying the meta-qualification condition based on the data calling request includes:

The index information corresponding to the corresponding type and satisfying the set qualification condition is obtained based on the data call request.
A data calling system, characterized in that the data elements in the original data set are stored in a corresponding target data set according to the data storage method according to any one of claims 1 to 10, and the target data set is stored in a computer device In the associated first storage device, the data calling system includes:

a user request obtaining module, configured to obtain a data calling request sent by the client, the data calling request at least including the type of the data to be called; and

A calling module, configured to obtain partial data in the target data set based on the data calling request to obtain the data to be called, where the partial data includes data in the target data set corresponding to the type to which the data to be called belongs Yuan.
The system according to claim 24, wherein the target data set has corresponding index information, and the index information at least includes meta identification information and storage location information corresponding to each data element in the target data set. , wherein the meta-identification information refers to the identification information of the corresponding data element; the data invocation request also includes meta-qualification conditions related to the meta-identification information;

The calling module includes:

an index information obtaining unit, configured to obtain index information corresponding to the corresponding type and satisfying the meta-qualification condition based on the data call request;

a segmented storage unit for dividing the target data set into a plurality of target data subsets at preset time intervals, and storing each target data subset respectively; and

The data element obtaining unit is configured to obtain the data element based on the storage location in the obtained index information.
The system according to claim 25, wherein the data elements in the target data set are arranged in chronological order, the meta identification information includes time information of the corresponding data elements; the meta qualification includes the data to be called corresponding time range.
The system according to claim 26, wherein the data calling system further comprises a synchronization module, and the synchronization module is configured to send the data to be called to the second storage device of the client.
The system according to claim 25, wherein the index information further comprises set identification information of the original data set corresponding to each data element in the target data set, wherein the set identification information refers to the identification information of the original data set ; The data call request also includes set qualifications related to the set identification information;

The index information acquisition unit is further used for:

The index information corresponding to the corresponding type and satisfying the set qualification condition is obtained based on the data call request.
A storage medium, characterized in that the storage medium is used to store computer instructions, and after a computer reads the computer instructions in the storage medium, it executes the data calling method according to any one of claims 18 to 23 .