CN110765075A - Storage method and equipment of automatic driving data - Google Patents
Storage method and equipment of automatic driving data Download PDFInfo
- Publication number
- CN110765075A CN110765075A CN201911015987.3A CN201911015987A CN110765075A CN 110765075 A CN110765075 A CN 110765075A CN 201911015987 A CN201911015987 A CN 201911015987A CN 110765075 A CN110765075 A CN 110765075A
- Authority
- CN
- China
- Prior art keywords
- file
- file storage
- storage system
- stored
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G07C5/00—Registering or indicating the working of vehicles
- G07C5/08—Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
- G07C5/0841—Registering performance data
- G07C5/085—Registering performance data using electronic data carriers
Abstract
The application discloses a storage method and equipment of automatic driving data, and relates to the field of automatic driving. The specific implementation scheme is as follows: receiving a file storage request sent by a terminal, wherein the file storage request carries configuration information of a file to be stored; determining a target file storage system for storing the file to be stored from at least two file storage systems according to the configuration information; and sending a notification message to the terminal, wherein the notification message is used for indicating the terminal to store the file to be stored in the target file storage system. Compared with the method for storing all files in one file storage system in the prior art, the scheme can store different files in the proper file storage system, and improves the stability and the performance of the file storage system. In addition, the terminal directly uploads the file to be stored to the target file storage system, transfer of the server is not needed, and storage efficiency is improved.
Description
Technical Field
The invention relates to a data storage technology, in particular to a method and equipment for storing automatic driving data.
Background
The data is the core driving force of the automatic driving technology, the data volume in the automatic driving field is huge, and a large amount of data generated by a client, a large amount of sensing data, a large amount of module operation log data and the like exist. How to uniformly manage and efficiently store the data is an important research topic in the field of automatic driving.
The existing storage platform is designed to only lay out a universal file storage system, and all files are stored in the file storage system. However, not all files are suitable for storage in current file storage systems, such as: small files are not suitable for being stored in the distributed file system (HDFS), and the name space of the HDFS is occupied. The stability and performance of a file storage system is clearly affected if all files are stored in one file storage system.
Disclosure of Invention
The invention provides a method and equipment for storing automatic driving data, which are used for improving the stability and the performance of a file storage system.
In a first aspect, the present invention provides a method for storing automatic driving data, including:
receiving a file storage request sent by a terminal, wherein the file storage request carries configuration information of a file to be stored;
determining a target file storage system for storing the file to be stored from at least two file storage systems according to the configuration information;
and sending a notification message to the terminal, wherein the notification message is used for indicating the terminal to store the file to be stored in the target file storage system.
By the scheme, different files can be stored in the appropriate file storage system, and the stability and the performance of the file storage system are improved. In addition, the terminal directly uploads the file to be stored to the target file storage system, transfer of the server is not needed, and storage efficiency is improved.
Optionally, before determining, according to the configuration information, a target file storage system for storing the file to be stored from the at least two file storage systems, the method further includes:
judging whether the configuration information carries specified information of a file storage system or not, wherein the specified information is used for indicating the type of the file storage system for storing the file to be stored;
the determining, according to the configuration information, a target file storage system for storing the file to be stored from at least two file storage systems includes:
and determining the target file storage system from at least two file storage systems according to the judgment result.
Optionally, the determining, according to the determination result, a target file storage system for storing the file to be stored from at least two file storage systems includes:
and if the configuration information carries the specified information, determining the target file storage system according to the specified information.
The target file storage system is determined according to the specified information, the file to be stored can be stored in the specified file system, so that the attributes of the file storage system and the stored file are more matched, and the performance of the file storage system is improved.
Optionally, the determining, according to the determination result, a target file storage system for storing the file to be stored from at least two file storage systems includes:
if the configuration information does not carry the specified information, acquiring the storage position of the file to be stored from the configuration information;
and determining a target file storage system for storing the file to be stored according to the storage position.
Optionally, the determining, according to the storage location, a target file storage system for storing the file to be stored includes:
according to the storage position, judging whether the file to be stored is stored in a first file storage system of the at least two file storage systems, wherein the first file storage system comprises: a distributed file system (HDFS);
and if not, determining the target file storage system according to the capacities of the at least two file storage systems.
Optionally, the determining the target file storage system according to the capacities of the at least two file storage systems includes:
if the capacity of only one file storage system in the at least two file storage systems is sufficient, determining the file storage system with the sufficient capacity in the at least two file storage systems as the target file storage system; therefore, the file to be stored can be stored in the file storage system with sufficient capacity, and the performance of the file storage system is improved.
And if the capacities of all the file storage systems in the at least two file storage systems are sufficient, acquiring the size of the file to be stored from the configuration information, and determining the target file storage system according to the size of the file to be stored.
Optionally, the determining the target file storage system according to the size of the file to be stored includes:
if the size of the file to be stored is larger than a preset threshold value, determining that the first file storage system is determined as the target file storage system;
if the size of the file to be stored is smaller than or equal to the preset threshold, determining that a second file storage system of the at least two file storage systems is the target file storage system, where the second file storage system includes: distributed storage system HBase. Therefore, the sizes of the file storage system and the stored files are matched, and the performance of the file storage system is improved.
Optionally, the determining the target file storage system according to the size of the file to be stored includes:
if the size of the file to be stored is smaller than or equal to a preset threshold, judging whether the file to be stored needs to interact with a front end according to the configuration information;
if so, determining that a third file storage system of the at least two file storage systems is the target file storage system, wherein the third file storage system comprises: a distributed file storage database MONGO.
Optionally, the notification message carries an identifier of the target file storage system and a generated storage address, and the notification message is used to instruct the terminal to store the file to be stored in the target file storage system according to the identifier of the target file storage system and the storage address.
Optionally, the method further includes:
receiving a storage completion message sent by a terminal, wherein the storage completion message is used for indicating to store the index information of the file to be stored;
and storing the index information of the file to be stored into a distributed file storage database MONGO according to the storage completion message.
Optionally, the method further includes:
receiving a retrieval keyword sent by a terminal;
retrieving to obtain the index information according to the retrieval key words;
and sending the index information to the terminal so that the terminal downloads the file to be stored from the target file storage system according to the index information. The terminal can download the file from the target file storage system according to the index information, so that the process of transferring the file at the server is omitted, and the file downloading efficiency is improved.
In a second aspect, the present invention provides an automatic driving data storage device, comprising:
the receiving module is used for receiving a file storage request sent by a terminal, wherein the file storage request carries configuration information of a file to be stored;
the determining module is used for determining a target file storage system for storing the file to be stored from at least two file storage systems according to the configuration information;
and the sending module is used for sending a notification message to the terminal, wherein the notification message is used for indicating the terminal to store the file to be stored in the target file storage system.
Optionally, the storage device further includes:
the determining module is used for determining whether the configuration information carries the specified information of the file storage system or not before the determining module determines a target file storage system for storing the file to be stored from at least two file storage systems according to the configuration information, wherein the specified information is used for indicating the type of the file storage system for storing the file to be stored;
correspondingly, the determining module is specifically configured to:
and determining the target file storage system from at least two file storage systems according to the judgment result.
Optionally, the determining module is specifically configured to:
and if the configuration information carries the specified information, determining the target file storage system according to the specified information.
If the configuration information does not carry the specified information, acquiring the storage position of the file to be stored from the configuration information;
and determining a target file storage system for storing the file to be stored according to the storage position.
Optionally, the determining module is specifically configured to:
according to the storage position, judging whether the file to be stored is stored in a first file storage system of the at least two file storage systems, wherein the first file storage system comprises: a distributed file system (HDFS);
and if not, determining the target file storage system according to the capacities of the at least two file storage systems.
Optionally, the determining module is specifically configured to:
if the capacity of only one file storage system in the at least two file storage systems is sufficient, determining the file storage system with the sufficient capacity in the at least two file storage systems as the target file storage system;
and if the capacities of all the file storage systems in the at least two file storage systems are sufficient, acquiring the size of the file to be stored from the configuration information, and determining the target file storage system according to the size of the file to be stored.
Optionally, the determining module is specifically configured to:
if the size of the file to be stored is larger than a preset threshold value, determining that the first file storage system is determined as the target file storage system;
if the size of the file to be stored is smaller than or equal to the preset threshold, determining that a second file storage system of the at least two file storage systems is the target file storage system, where the second file storage system includes: distributed storage system HBase.
Optionally, the determining module is specifically configured to:
if the size of the file to be stored is smaller than or equal to a preset threshold, judging whether the file to be stored needs to interact with a front end according to the configuration information;
if so, determining that a third file storage system of the at least two file storage systems is the target file storage system, wherein the third file storage system comprises: a distributed file storage database MONGO.
Optionally, the notification message carries an identifier of the target file storage system and a generated storage address, and the notification message is used to instruct the terminal to store the file to be stored in the target file storage system according to the identifier of the target file storage system and the storage address.
Optionally, the storage device further includes: a storage module;
the receiving module is further configured to: receiving a storage completion message sent by a terminal, wherein the storage completion message is used for indicating to store the index information of the file to be stored;
the storage module is used for: and storing the index information of the file to be stored into a distributed file storage database MONGO according to the storage completion message.
Optionally, the apparatus further comprises: a retrieval module;
the receiving module is also used for receiving the retrieval key words sent by the terminal;
the retrieval module is used for retrieving to obtain the index information according to the retrieval key words;
the sending module is further configured to send the index information to the terminal, so that the terminal downloads the file to be stored from the target file storage system according to the index information.
In a third aspect, the present invention provides an electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described method of storing autopilot data.
In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the above-described storage method of automatic driving data.
In a fifth aspect, the present invention provides a storage platform, which includes the above electronic device and at least two file storage systems.
Compared with the method for storing all files in one file storage system in the prior art, the method and the device for storing the automatic driving data can store different files in the proper file storage system, and improve the stability and the performance of the file storage system. In addition, the terminal directly uploads the file to be stored to the target file storage system, transfer of the server is not needed, and storage efficiency is improved.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
FIG. 1 is an alternative application scenario diagram of the method of storing autopilot data provided by the present invention;
FIG. 2 is a schematic flow chart illustrating a first embodiment of a method for storing autopilot data according to the present invention;
FIG. 3 is a flowchart illustrating a second embodiment of a method for storing autopilot data according to the present invention;
FIG. 4 is a schematic flow chart of a second embodiment of the present invention;
FIG. 5 is a schematic flow chart illustrating a third embodiment of a method for storing automatic driving data according to the present invention;
FIG. 6 is a schematic diagram of an embodiment of an automatic driving data storage device according to the present invention;
fig. 7 is a schematic diagram of a hardware structure of the electronic device provided by the present invention.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In terms of file storage, the existing storage platform is designed to only lay out a general file storage system, and all files are stored in the file storage system. When a user needs to store and download files, the files must be transferred to a server provided by a storage platform, and then the files are transmitted to a file storage system or a local place by the server. However, the design of such a storage platform has the following problems: not all files are suitable for storage in current file storage systems, such as: small files are not suitable for being stored in the distributed file system (HDFS), and the name space of the HDFS is occupied. The stability and performance of a file storage system is clearly affected if all files are stored in one file storage system. In addition, the process of transferring files at the server side affects the overall storage and downloading efficiency.
Based on the technical problems in the prior art, the invention provides a method for storing autopilot data, which includes determining a target file storage system for storing a file to be stored from at least two file storage systems according to configuration information of the file to be stored carried in a file storage request after receiving the file storage request sent by a terminal, and then sending a notification message to the terminal so that the terminal directly stores the file to be stored into the target file storage system according to the notification message. Compared with the method for storing all files in one file storage system in the prior art, the scheme of the invention can store the files with different configurations in the proper file storage system, thereby improving the stability and the performance of the file storage system. Moreover, the scheme of the invention allows the terminal to directly upload the file to be stored to the target file storage system without the transfer of the server, thereby improving the storage efficiency.
Fig. 1 is an optional application scenario diagram of the storage method of the automatic driving data provided by the present invention. The application scenario shown in fig. 1 includes a terminal, a server, and at least two file storage systems.
The terminal is provided with a client, and a user can finish storage and downloading of files through the client. Specifically, when a user needs to store a file, a file storage request can be triggered through the client, and the client further sends the file storage request to the server, so that the server determines a target file storage system suitable for storing the file to be stored from at least two file storage systems by executing the scheme of the invention, and returns a notification message to the client, so that the client directly stores the file to be stored into the target file storage system according to the notification message. On the other hand, when a user needs to download a file, the user can input a search keyword through the client, the client further sends the search keyword to the server, so that the server can search according to the search keyword to obtain corresponding index information, and after the server returns the index information to the client, the client can directly download the file from a corresponding file storage system according to the index information. The file storage process realizes the classified storage of the files and improves the stability and the performance of each file storage system; the retrieval of the file is realized through the index information in the file downloading process, so that the retrieval efficiency and convenience are improved; and the storage and downloading processes are directly and interactively completed by the client and the file storage system, the transfer of the server is not needed, and the efficiency of the storage and downloading processes is greatly improved.
Optionally, the terminal may be a smart phone, a tablet computer, a computer, or the like, which may implement interaction with a user. The server can be a cloud server or an entity server. At least two file storage systems are illustrated in fig. 1 as file storage system 1, file storage system 2 and file storage system N. The at least two file storage systems may include: the distributed file system HDFS, the distributed storage system HBase, the distributed file storage database MONGO, and the like, and optionally, the distributed file storage database MONGO may also be distributed on the server.
The following describes the technical solutions of the present invention and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present invention will be described below with reference to the accompanying drawings.
Fig. 2 is a schematic flow chart of a first embodiment of a method for storing automatic driving data according to the present invention. The storage method of the automatic driving data provided by the embodiment can be executed by the server in the scene diagram shown in fig. 1. As shown in fig. 2, the method for storing the automatic driving data according to the present embodiment includes:
s201, receiving a file storage request sent by a terminal, wherein the file storage request carries configuration information of a file to be stored.
S202, according to the configuration information, determining a target file storage system for storing the file to be stored from at least two file storage systems.
Specifically, when a user needs to store a file, a file storage request can be triggered through a client installed on a terminal, and the file storage request carries configuration information of the file to be stored. The configuration information may include: the file storage system comprises the specific information of the file storage system, the storage position of the file to be stored, the capacity of each of at least two file storage systems, the size of the file to be stored and the like. And the client sends a file storage request containing the configuration information to the server, so that the server selects a system suitable for storing the file to be stored from at least two file storage systems as a target file storage system according to the configuration information.
S203, sending a notification message to the terminal, wherein the notification message is used for indicating the terminal to store the file to be stored in the target file storage system.
Specifically, after determining the target file storage system in S202, the server sends a notification message to the terminal, where the notification message may carry an identifier of the target file storage system and the generated storage address, and after receiving the notification message, the terminal may store the file to be stored in the target file storage system according to the identifier of the target file storage system and the storage address.
The embodiment provides a method for storing automatic driving data, which includes determining a target file storage system for storing a file to be stored from at least two file storage systems according to configuration information of the file to be stored, which is carried in a file storage request, after receiving the file storage request sent by a terminal, and then sending a notification message to the terminal, so that the terminal directly stores the file to be stored into the target file storage system according to the notification message. Compared with the method for storing all files in one file storage system in the prior art, the method of the embodiment can store different files in the appropriate file storage system, and improves the stability and performance of the file storage system. In addition, the terminal directly uploads the file to be stored to the target file storage system, transfer of the server is not needed, and storage efficiency is improved.
Fig. 3 is a flowchart illustrating a second embodiment of the storage method of the automatic driving data according to the present invention. This embodiment describes in detail how the target file storage system can be determined according to the configuration information in the above-mentioned embodiment S202. Specifically, the method for storing the automatic driving data provided by the embodiment includes:
s301, receiving a file storage request sent by a terminal, wherein the file storage request carries configuration information of a file to be stored.
As described above, the configuration information may include: the file storage system comprises the specific information of the file storage system, the storage position of the file to be stored, the capacity of each of at least two file storage systems, the size of the file to be stored and the like.
S302, judging whether the configuration information carries the specified information of the file storage system. The specified information is used for indicating the type of the file storage system for storing the file to be stored.
After the judgment result is obtained, determining the target file storage system from at least two file storage systems according to the judgment result, specifically, if the judgment result is yes, executing step S303; if the determination result is negative, S304 is executed.
S303, determining the target file storage system according to the specified information.
Taking sensor data on an automatic driving automobile as an example, assuming that the automatic driving automobile runs for 4 hours, wherein only 2 minutes of data has abnormality, a user only wants to obtain the 2 minutes of abnormal data when using the data, but does not want to take 4 hours of data at a time, the data of the type needs to be stored in a distributed storage system HBase, and the data of the type can be explicitly configured with specified information during configuration, wherein the specified information is used for indicating that the type of a file storage system for storing files to be stored is HBase. After receiving the file storage request of the data, the server directly determines the HBase as a target file storage system under the condition that the configuration information is judged to have the specified information, generates a storage address of the HBase and sends the storage address to the terminal, so that the terminal directly stores the file to be stored into the HBase according to the storage address.
And if the specified information indicates that the type of the file storage system storing the file to be stored is the HDFS, the server directly determines the HDFS as the target file storage system.
S304, obtaining the storage position of the file to be stored from the configuration information.
And after the storage position of the file to be stored is obtained, determining a target file storage system for storing the file to be stored according to the storage position. Specifically, the implementation of determining the target file storage system according to the storage location includes S305.
S305, judging whether the file to be stored is stored in the first file storage system according to the storage position.
If so, the file to be stored already exists in the first file storage system, the file does not need to be stored again, and the storage process is not continued; and if not, determining the target file storage system according to the capacities of the at least two file storage systems.
Optionally, the first file storage system may be a distributed file system HDFS.
Specifically, the implementation manner of determining the target file storage system according to the capacity of the file storage system includes: if the capacity of only one file storage system of the at least two file storage systems is sufficient, executing S306; if the capacities of all the file storage systems of the at least two file storage systems are sufficient, S307 is executed.
S306, determining the file storage system with sufficient capacity in the at least two file storage systems as the target file storage system.
S307, the size of the file to be stored is obtained from the configuration information.
After the size of the file to be stored is obtained, comparing the size of the file to be stored with a preset threshold, and if the size of the file to be stored is larger than the preset threshold, executing S308; if the size of the file to be stored is smaller than or equal to the preset threshold, S309 is executed.
S308, determining that the first file storage system is determined to be the target file storage system.
The first file storage system is an HDFS.
S309, determining that the second file storage system is the target file storage system.
Wherein the second file storage system is HBase.
Fig. 4 is another flow chart of the second embodiment. Referring to fig. 4, after the size of the file to be stored is acquired in S307, when it is determined that the size of the file to be stored is smaller than or equal to the preset threshold, S3010 is executed.
S3010, judging whether the file to be stored needs to interact with a front end according to the configuration information.
If yes, S3011 is executed.
S3011, determining that a third file storage system of the at least two file storage systems is the target file storage system.
Wherein the third file storage system may be a distributed file storage database MONGO.
After determining a target file storage system storing the file to be stored from at least two file storage systems through S302-S3011, S3012-S3014 are executed.
S3012, sending a notification message to the terminal.
And the notification message is used for indicating the terminal to store the file to be stored in the target file storage system.
And S3013, receiving a storage completion message sent by the terminal.
And the storage completion message is used for indicating that the index information of the file to be stored is stored.
S3014, storing the index information of the file to be stored into a distributed file storage database MONGO according to the storage completion message.
Specifically, after receiving the storage completion message sent by the terminal, the server may determine that the storage of the file to be stored is completed, and at this time, may store the index information of the file to be stored in the distributed file storage database MONGO.
Specifically, the storage structure design of the index information can be divided into 4 parts: rowkey, sys, meta, and data.
The data in sys mainly includes data automatically calculated by the system, such as: creation time, update user, request host, request ip, file type, and file origin path. "creation time" refers to the time a file is placed in the storage system. Wherein "update time" refers to the time at which the file was changed. "update user" refers to the user name that changes the file. "request host" refers to changing the host name of a file. "request ip" refers to the host ip that changes the file. The "file type" refers to a local file, a character string, or a file already stored on the HDFS. "file origin path" refers to the machine path in which the local file is located. The data in meta is user-defined data that describes some characteristics of the file to be stored. The data in the data is mainly used for recording storage clusters and paths of the files and the type of the file storage system in which the files are stored. rowkey is a key that the system globally uniquely identifies a piece of data. The design of the storage structure can realize that: 1. all data of automatic driving can be uniformly stored; 2. flexible expansion can be performed for different types of data; 3. the distributed storage system can realize horizontal capacity expansion to support large-scale data.
The storage method of the automatic driving data provided by the embodiment describes in detail the implementation manner of determining the target file storage system by the server according to the configuration information, and the method of determining the target file storage system by the server according to the configuration information can store the file to be stored in the appropriate file storage system, thereby improving the stability and performance of the file storage system.
Fig. 5 is a schematic flow chart of a third embodiment of the storage method of the automatic driving data provided by the present invention. On the basis of the above-described embodiment, the present embodiment describes in detail the downloading process of the automatic driving data. The storage method of the automatic driving data provided by the embodiment comprises the following steps:
s501, receiving a file storage request sent by a terminal, wherein the file storage request carries configuration information of a file to be stored.
S502, according to the configuration information, determining a target file storage system for storing the file to be stored from at least two file storage systems.
S503, sending a notification message to the terminal, wherein the notification message is used for instructing the terminal to store the file to be stored in the target file storage system.
S504, receiving a storage completion message sent by the terminal, wherein the storage completion message is used for indicating that the index information of the file to be stored is stored.
And S505, storing the index information of the file to be stored into a distributed file storage database MONGO according to the storage completion message.
The implementation manners of the above S501 to S505 can refer to the above embodiments, and the present invention is not described herein again.
And S506, receiving the search keyword sent by the terminal.
And S507, retrieving to obtain the index information according to the retrieval key words.
And S508, sending the index information to the terminal so that the terminal downloads the file to be stored from the target file storage system according to the index information.
Compared with the prior art that the file needs to be downloaded through the server side transfer, in the embodiment, the terminal directly downloads the file from the target file storage system according to the index information returned by the server, so that the process of transferring the file by the server side is omitted, and the file downloading efficiency is improved.
According to the storage method of the automatic driving data, the downloading process of the automatic driving data is described, and after the server returns the index information to the terminal, the terminal can automatically download the file from the target file storage system according to the index information, so that the process of transferring the file by the server is omitted, and the file downloading efficiency is improved.
Fig. 6 is a schematic structural diagram of an automatic driving data storage device according to the present invention. As shown in fig. 6, the present invention provides an automatic driving data storage device, including:
a receiving module 601, configured to receive a file storage request sent by a terminal, where the file storage request carries configuration information of a file to be stored;
a determining module 602, configured to determine, according to the configuration information, a target file storage system that stores the file to be stored from at least two file storage systems;
a sending module 603, configured to send a notification message to the terminal, where the notification message is used to instruct the terminal to store the file to be stored in the target file storage system.
Optionally, the storage device further includes:
a determining module 604, configured to determine, by the determining module, whether the configuration information carries designation information of a file storage system before a target file storage system storing the file to be stored is determined from at least two file storage systems according to the configuration information, where the designation information is used to indicate a type of the file storage system storing the file to be stored;
correspondingly, the determining module 602 is specifically configured to:
and determining the target file storage system from at least two file storage systems according to the judgment result.
Optionally, the determining module 602 is specifically configured to:
and if the configuration information carries the specified information, determining the target file storage system according to the specified information.
If the configuration information does not carry the specified information, acquiring the storage position of the file to be stored from the configuration information;
and determining a target file storage system for storing the file to be stored according to the storage position.
Optionally, the determining module 602 is specifically configured to:
according to the storage position, judging whether the file to be stored is stored in a first file storage system of the at least two file storage systems, wherein the first file storage system comprises: a distributed file system (HDFS);
and if not, determining the target file storage system according to the capacities of the at least two file storage systems.
Optionally, the determining module 602 is specifically configured to:
if the capacity of only one file storage system in the at least two file storage systems is sufficient, determining the file storage system with the sufficient capacity in the at least two file storage systems as the target file storage system;
and if the capacities of all the file storage systems in the at least two file storage systems are sufficient, acquiring the size of the file to be stored from the configuration information, and determining the target file storage system according to the size of the file to be stored.
Optionally, the determining module 602 is specifically configured to:
if the size of the file to be stored is larger than a preset threshold value, determining that the first file storage system is determined as the target file storage system;
if the size of the file to be stored is smaller than or equal to the preset threshold, determining that a second file storage system of the at least two file storage systems is the target file storage system, where the second file storage system includes: distributed storage system HBase.
Optionally, the determining module 602 is specifically configured to:
if the size of the file to be stored is smaller than or equal to a preset threshold, judging whether the file to be stored needs to interact with a front end according to the configuration information;
if so, determining that a third file storage system of the at least two file storage systems is the target file storage system, wherein the third file storage system comprises: a distributed file storage database MONGO.
Optionally, the notification message carries an identifier of the target file storage system and a generated storage address, and the notification message is used to instruct the terminal to store the file to be stored in the target file storage system according to the identifier of the target file storage system and the storage address.
Optionally, the storage device further includes: a storage module 605;
the receiving module is further configured to: receiving a storage completion message sent by a terminal, wherein the storage completion message is used for indicating to store the index information of the file to be stored;
the storage module is used for: and storing the index information of the file to be stored into a distributed file storage database MONGO according to the storage completion message.
Optionally, the apparatus further comprises: a retrieval module 606;
the receiving module is also used for receiving the retrieval key words sent by the terminal;
the retrieval module is used for retrieving to obtain the index information according to the retrieval key words;
the sending module is further configured to send the index information to the terminal, so that the terminal downloads the file to be stored from the target file storage system according to the index information.
The storage device for automatic driving data provided in this embodiment may be used to execute the steps in any of the above method embodiments, and the implementation principle and technical effect are similar, which are not described herein again.
According to an embodiment of the present application, an electronic device and a readable storage medium are also provided.
As shown in fig. 7, it is a block diagram of an electronic device of a storage method of automatic driving data according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 7, the electronic apparatus includes: one or more processors 701, a memory 702, and interfaces for connecting the various components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). In fig. 7, one processor 701 is taken as an example.
The memory 702 is a non-transitory computer readable storage medium as provided herein. Wherein the memory stores instructions executable by at least one processor to cause the at least one processor to perform the method of storing autopilot data provided herein. The non-transitory computer-readable storage medium of the present application stores computer instructions for causing a computer to execute the storing method of the automatic driving data provided by the present application.
The memory 702, which is a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions/modules corresponding to the storage method of the automatic driving data in the embodiment of the present application (for example, the receiving module 601, the determining module 602, the transmitting module 603, the determining module 604, the storing module 605, and the retrieving module 606 shown in fig. 6). The processor 701 executes various functional applications of the server and data processing, i.e., implements the storage method of the automatic driving data in the above-described method embodiment, by running the non-transitory software programs, instructions, and modules stored in the memory 702.
The memory 702 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to a storage method of the automatic driving data, and the like. Further, the memory 702 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 702 optionally includes memory located remotely from processor 701, which may be connected via a network to electronics that enable storage of autopilot data. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected by a bus or other means, and fig. 7 illustrates an example of a connection by a bus.
The input device 703 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic apparatus that enables storage of the autopilot data, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointer, one or more mouse buttons, a track ball, a joystick, and the like. The output devices 704 may include a display device, auxiliary lighting devices (e.g., LEDs), and tactile feedback devices (e.g., vibrating motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme of the embodiment of the application, different files can be stored in the appropriate file storage system, and compared with the method for storing all the files in one file storage system in the prior art, the stability and the performance of the file storage system are improved. In addition, the terminal directly uploads the file to be stored to the target file storage system, transfer of the server is not needed, and storage efficiency is improved.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (15)
1. A method of storing autonomous driving data, comprising:
receiving a file storage request sent by a terminal, wherein the file storage request carries configuration information of a file to be stored;
determining a target file storage system for storing the file to be stored from at least two file storage systems according to the configuration information;
and sending a notification message to the terminal, wherein the notification message is used for indicating the terminal to store the file to be stored in the target file storage system.
2. The method according to claim 1, wherein before determining a target file storage system for storing the file to be stored from at least two file storage systems according to the configuration information, further comprising:
judging whether the configuration information carries specified information of a file storage system or not, wherein the specified information is used for indicating the type of the file storage system for storing the file to be stored;
the determining, according to the configuration information, a target file storage system for storing the file to be stored from at least two file storage systems includes:
and determining the target file storage system from at least two file storage systems according to the judgment result.
3. The method of claim 2,
the determining, according to the determination result, a target file storage system storing the file to be stored from at least two file storage systems includes:
and if the configuration information carries the specified information, determining the target file storage system according to the specified information.
4. The method of claim 2,
the determining, according to the determination result, a target file storage system storing the file to be stored from at least two file storage systems includes:
if the configuration information does not carry the specified information, acquiring the storage position of the file to be stored from the configuration information;
and determining a target file storage system for storing the file to be stored according to the storage position.
5. The method according to claim 4, wherein determining a target file storage system for storing the file to be stored according to the storage location comprises:
according to the storage position, judging whether the file to be stored is stored in a first file storage system of the at least two file storage systems, wherein the first file storage system comprises: a distributed file system (HDFS);
and if not, determining the target file storage system according to the capacities of the at least two file storage systems.
6. The method of claim 5, wherein determining the target file storage system based on the capacities of the at least two file storage systems comprises:
if the capacity of only one file storage system in the at least two file storage systems is sufficient, determining the file storage system with the sufficient capacity in the at least two file storage systems as the target file storage system;
and if the capacities of all the file storage systems in the at least two file storage systems are sufficient, acquiring the size of the file to be stored from the configuration information, and determining the target file storage system according to the size of the file to be stored.
7. The method according to claim 6, wherein determining the target file storage system according to the size of the file to be stored comprises:
if the size of the file to be stored is larger than a preset threshold value, determining that the first file storage system is determined as the target file storage system;
if the size of the file to be stored is smaller than or equal to the preset threshold, determining that a second file storage system of the at least two file storage systems is the target file storage system, where the second file storage system includes: distributed storage system HBase.
8. The method according to claim 6, wherein determining the target file storage system according to the size of the file to be stored comprises:
if the size of the file to be stored is smaller than or equal to a preset threshold, judging whether the file to be stored needs to interact with a front end according to the configuration information;
if so, determining that a third file storage system of the at least two file storage systems is the target file storage system, wherein the third file storage system comprises: a distributed file storage database MONGO.
9. The method according to any one of claims 1 to 8, wherein the notification message carries an identifier of the target file storage system and a generated storage address, and the notification message is used to instruct the terminal to store the file to be stored in the target file storage system according to the identifier of the target file storage system and the storage address.
10. The method of claim 9, further comprising:
receiving a storage completion message sent by a terminal, wherein the storage completion message is used for indicating to store the index information of the file to be stored;
and storing the index information of the file to be stored into a distributed file storage database MONGO according to the storage completion message.
11. The method of claim 9, further comprising:
receiving a retrieval keyword sent by a terminal;
retrieving to obtain the index information according to the retrieval key words;
and sending the index information to the terminal so that the terminal downloads the file to be stored from the target file storage system according to the index information.
12. An automatic driving data storage device, comprising:
the receiving module is used for receiving a file storage request sent by a terminal, wherein the file storage request carries configuration information of a file to be stored;
the determining module is used for determining a target file storage system for storing the file to be stored from at least two file storage systems according to the configuration information;
and the sending module is used for sending a notification message to the terminal, wherein the notification message is used for indicating the terminal to store the file to be stored in the target file storage system.
13. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.
14. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-11.
15. A storage platform comprising the electronic device of claim 13 and at least two file storage systems.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911015987.3A CN110765075A (en) | 2019-10-24 | 2019-10-24 | Storage method and equipment of automatic driving data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911015987.3A CN110765075A (en) | 2019-10-24 | 2019-10-24 | Storage method and equipment of automatic driving data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110765075A true CN110765075A (en) | 2020-02-07 |
Family
ID=69333310
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911015987.3A Pending CN110765075A (en) | 2019-10-24 | 2019-10-24 | Storage method and equipment of automatic driving data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110765075A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111367857A (en) * | 2020-03-03 | 2020-07-03 | 中国联合网络通信集团有限公司 | Data storage method and device, FTP server and storage medium |
| CN112416858A (en) * | 2020-11-09 | 2021-02-26 | 深圳市珍爱捷云信息技术有限公司 | Document storage method and device, electronic equipment and computer readable storage medium |
| CN114040223A (en) * | 2021-11-05 | 2022-02-11 | 湖北亿咖通科技有限公司 | Image processing method and system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101944124A (en) * | 2010-09-21 | 2011-01-12 | 卓望数码技术（深圳）有限公司 | Distributed file system management method, device and corresponding file system |
| CN106886610A (en) * | 2017-04-11 | 2017-06-23 | 广东浪潮大数据研究有限公司 | The file management method and device of a kind of distributed file system |
| US10360076B1 (en) * | 2016-10-06 | 2019-07-23 | Sprint Communications Company L.P. | Prioritized rebalancing of distributed file system |
-
2019
- 2019-10-24 CN CN201911015987.3A patent/CN110765075A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101944124A (en) * | 2010-09-21 | 2011-01-12 | 卓望数码技术（深圳）有限公司 | Distributed file system management method, device and corresponding file system |
| US10360076B1 (en) * | 2016-10-06 | 2019-07-23 | Sprint Communications Company L.P. | Prioritized rebalancing of distributed file system |
| CN106886610A (en) * | 2017-04-11 | 2017-06-23 | 广东浪潮大数据研究有限公司 | The file management method and device of a kind of distributed file system |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111367857A (en) * | 2020-03-03 | 2020-07-03 | 中国联合网络通信集团有限公司 | Data storage method and device, FTP server and storage medium |
| CN112416858A (en) * | 2020-11-09 | 2021-02-26 | 深圳市珍爱捷云信息技术有限公司 | Document storage method and device, electronic equipment and computer readable storage medium |
| CN114040223A (en) * | 2021-11-05 | 2022-02-11 | 湖北亿咖通科技有限公司 | Image processing method and system |
| CN114040223B (en) * | 2021-11-05 | 2023-11-24 | 亿咖通(湖北)技术有限公司 | Image processing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111753997B (en) | Distributed training method, system, device and storage medium | |
| CN111258609A (en) | Upgrading method and device of Kubernetes cluster, electronic equipment and medium | |
| CN111045985B (en) | File storage processing method, server, electronic device and storage medium | |
| CN112073994A (en) | Method and device for over-the-air upgrading, electronic equipment and readable storage medium | |
| CN110765075A (en) | Storage method and equipment of automatic driving data | |
| CN112000394B (en) | Method, apparatus, device and storage medium for accessing applet | |
| CN111835856A (en) | File downloading method, device, equipment and storage medium | |
| CN111752960B (en) | Data processing method and device | |
| CN111913884A (en) | Distributed test method, device, equipment, system and readable storage medium | |
| CN112583904A (en) | File uploading method, device, equipment and storage medium | |
| CN111966471B (en) | Access method, device, electronic equipment and computer storage medium | |
| CN112565356B (en) | Data storage method and device and electronic equipment | |
| CN111796851A (en) | Client base library updating method and device, electronic equipment and storage medium | |
| CN110750419B (en) | Offline task processing method and device, electronic equipment and storage medium | |
| CN111767442A (en) | Data updating method and device, search server, terminal and storage medium | |
| CN112379945B (en) | Method, apparatus, device and storage medium for running application | |
| CN111782633B (en) | Data processing method and device and electronic equipment | |
| CN113163257B (en) | Screen projection method, device, equipment and storage medium | |
| CN112379954A (en) | Data processing method, device and equipment of application program and storage medium | |
| CN111340222A (en) | Neural network model searching method and device and electronic equipment | |
| CN111026438A (en) | Extraction method, device, equipment and medium for key information of small program package and page | |
| CN111459887B (en) | Resource screening method and device, electronic equipment and storage medium | |
| CN110531952B (en) | Voice skill processing method, device and equipment | |
| US20220284902A1 (en) | Delay estimation method and apparatus for smart rearview mirror, and electronic device | |
| CN112016020B (en) | Information processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20211026 Address after: 105 / F, building 1, No. 10, Shangdi 10th Street, Haidian District, Beijing 100085 Applicant after: Apollo Intelligent Technology (Beijing) Co.,Ltd. Address before: 2 / F, baidu building, 10 Shangdi 10th Street, Haidian District, Beijing 100085 Applicant before: BEIJING BAIDU NETCOM SCIENCE AND TECHNOLOGY Co.,Ltd. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200207 |