CN113742301A - Data storage method, system, electronic equipment and storage medium - Google Patents

Data storage method, system, electronic equipment and storage medium Download PDF

Info

Publication number
CN113742301A
CN113742301A CN202110862681.2A CN202110862681A CN113742301A CN 113742301 A CN113742301 A CN 113742301A CN 202110862681 A CN202110862681 A CN 202110862681A CN 113742301 A CN113742301 A CN 113742301A
Authority
CN
China
Prior art keywords
data
json file
snowball
comparison
storage
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110862681.2A
Other languages
Chinese (zh)
Inventor
李森林
周风明
朱敦华
刘强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan Kotei Informatics Co Ltd
Original Assignee
Wuhan Kotei Informatics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan Kotei Informatics Co Ltd filed Critical Wuhan Kotei Informatics Co Ltd
Priority to CN202110862681.2A priority Critical patent/CN113742301A/en
Publication of CN113742301A publication Critical patent/CN113742301A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/172Caching, prefetching or hoarding of files
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614Improving the reliability of storage systems
    • G06F3/0619Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Human Computer Interaction (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention provides a data storage method, a system, an electronic device and a storage medium, wherein the method comprises the following steps: generating a first json file based on local data pre-uploaded by a vehicle end, and uploading the first json file to Snowball equipment; after the data in the Snowball equipment is sent to the data center, the data center transmits the data in the Snowball equipment to a storage bucket; acquiring data in a storage bucket by calling a predetermined Python tool kit, and generating a second json file corresponding to the storage bucket data; and comparing the first json file with the second json file, judging whether the file data are consistent, generating a third json file based on the comparison result, marking whether the data comparison is normal in the third json file, and retransmitting abnormal data by comparison. Therefore, the vehicle data transmission and storage efficiency can be improved, and the consistency of the stored data can be effectively guaranteed.

Description

Data storage method, system, electronic equipment and storage medium
Technical Field
The invention belongs to the field of data processing, and particularly relates to a data storage method, a data storage system, electronic equipment and a storage medium.
Background
In the process of driving of a general vehicle, an automatic driving system needs to collect driving data of the vehicle in real time for storage, analysis, backup and the like. In the data storage process, the amount of data to be transmitted and stored is large, and the conventional hard disk storage is difficult to meet the requirement of transmission speed, so that the storage efficiency is low.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data storage method, a data storage system, an electronic device, and a storage medium, which are used to solve the problem of low storage efficiency of an existing hard disk.
In a first aspect of the embodiments of the present invention, a data storage method is provided, including:
generating a first json file based on local data pre-uploaded by a vehicle end, and uploading the first json file to Snowball equipment;
after the data in the Snowball equipment is sent to the data center, the data center transmits the data in the Snowball equipment to a storage bucket;
acquiring data in a storage bucket by calling a predetermined Python tool kit, and generating a second json file corresponding to the storage bucket data;
and comparing the first json file with the second json file, judging whether the file data are consistent, generating a third json file based on the comparison result, marking whether the data comparison is normal in the third json file, and retransmitting abnormal data by comparison.
In a second aspect of embodiments of the present invention, there is provided a data storage system, including:
the uploading module is used for generating a first json file based on local data pre-uploaded by the vehicle end and uploading the first json file to the Snowball equipment;
the data transmission module is used for transmitting the data in the Snowball equipment to the data center, and then the data center transmits the data in the Snowball equipment to the storage bucket;
the generating module is used for acquiring data in the storage bucket by calling a preset Python tool package and generating a second json file corresponding to the data of the storage bucket;
and the comparison retransmission module is used for comparing the first json file with the second json file, judging whether the file data are consistent, generating a third json file based on the comparison result, marking whether the data comparison is normal in the third json file, and retransmitting abnormal data by comparison.
In a third aspect of the embodiments of the present invention, there is provided an apparatus, including a memory, a processor, and a computer program stored in the memory and executable by the processor, where the processor executes the computer program to implement the steps of the method according to the first aspect of the embodiments of the present invention.
In a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, which stores a computer program, which when executed by a processor implements the steps of the method provided by the first aspect of the embodiments of the present invention.
In the embodiment of the invention, the local vehicle data are uploaded to the cloud for storage, so that the data transmission and storage speed can be improved, the TB-level data storage requirements can be met, and meanwhile, the json file comparison is carried out on the uploaded data, so that the data consistency can be ensured, and the data storage is accurate and efficient.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a schematic flowchart of a data storage method according to an embodiment of the present invention;
fig. 2 is another schematic flow chart of a data storage method according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a data storage system according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the embodiments described below are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be understood that the terms "comprises" and "comprising," when used in this specification or claims and in the accompanying drawings, are intended to cover a non-exclusive inclusion, such that a process, method or system, or apparatus that comprises a list of steps or elements is not limited to the listed steps or elements. In addition, "first" and "second" are used to distinguish different objects, and are not used to describe a specific order.
Referring to fig. 1, a flow chart of a data storage method according to an embodiment of the present invention includes:
s101, generating a first json file based on local data pre-uploaded by a vehicle end, and uploading the first json file to Snowball equipment;
before uploading driving data collected by a vehicle to a cloud end, a json file needs to be generated according to collected local data. The json file is a lightweight data exchange format, can represent or store data by adopting a text format, and can improve the data transmission efficiency by transmitting local data by adopting the json format.
The Snowball equipment is data storage intermediate equipment provided by the Amazon cloud service, can transfer local mass data into the Snowball, and transmits the mass data to the cloud service center through the Snowball.
The first json file at least contains the file name and the size of each piece of local data.
S102, after the data in the Snowball equipment are sent to a data center, the data center transmits the data in the Snowball equipment to a storage bucket;
and sending the Snowball equipment to an Amazon network service (AWS) center, and transmitting the data in the Snowball to an AWS cloud storage bucket by the data center. The Bucket (Bucket) is a carrier of the object and used for storing object data, and a user can create the Bucket and upload the object data according to an API provided by the Amazon cloud service.
Optionally, as shown in fig. 2, a Snowball command calling the AWSSCLL is executed, and the data storage state is acquired through the cloud service interface. And running a snowball interface program of the AWSCLL, acquiring the progress state of the snowball task in real time, and if the progress state is the completed state, indicating that the snowball operation is completed, namely the data transmission task on the snowball equipment is completed.
Wherein, awscll (amazon Web service command Line interface) is a command Line tool for managing AWS (amazon Web service) service, and uses corresponding service command to operate and access AWS service.
S103, acquiring data in a storage bucket by calling a preset Python tool kit, and generating a second json file corresponding to the storage bucket data;
and calling a boto3 toolkit in Python to read the cloud storage bucket data. For data stored in the AWS cloud bucket, the corresponding cloud data is obtained through the bato3 software development kit.
For vehicle end data stored in the cloud storage barrel, corresponding stored data are obtained through a reading command of the tool kit to generate a json file, and data consistency comparison is facilitated.
S104, comparing the first json file with the second json file, judging whether file data are consistent, generating a third json file based on a comparison result, marking whether data comparison is normal in the third json file, and retransmitting abnormal data through comparison.
The first json file is a json file generated according to local data, the second json file is a json file generated according to cloud storage data, the first json file and the second json file are compared, and whether all data of the first json file exist in the second json file or not and the sizes of the data are consistent are judged. Based on the comparison results, a corresponding json file may be generated.
Adding a first identifier for the abnormal comparison data in the third json file, and marking the normal comparison data as a second identifier; and acquiring a third json file, and when detecting a first identifier corresponding to the abnormal data, manually retransmitting the abnormal data.
In this embodiment, upload the data of gathering in the real vehicle road test and store and handle in the high in the clouds, promoted data transmission and processing speed, reduced the hard disk and used and the maintenance cost, conveniently carry out the efficient to large batch TB level data and handle.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
Fig. 3 is a schematic structural diagram of a data storage system according to an embodiment of the present invention, where the system includes:
the uploading module 310 is configured to generate a first json file based on local data pre-uploaded by a vehicle end, and upload the first json file to Snowball equipment;
the first json file at least contains the file name and the size of each piece of local data.
The data transmission module 320 is configured to transmit data in the Snowball equipment to the data center, and then the data center transmits the data in the Snowball equipment to the storage bucket;
optionally, a Snowball command for calling the AWSSCLL is executed, and the data storage state is acquired through the cloud service interface.
The generating module 330 is configured to obtain data in a bucket by calling a predetermined Python toolkit, and generate a second json file corresponding to the bucket data;
and acquiring data in the cloud storage bucket by calling the boto3 tool package in the Python.
And the comparison retransmission module 340 is configured to compare the first json file with the second json file, determine whether file data are consistent, generate a third json file based on the comparison result, mark whether data comparison is normal in the third json file, and retransmit abnormal data by comparison.
Specifically, a first identifier is added to the abnormal comparison data in the third json file, and the normal comparison data is marked as a second identifier; and acquiring a third json file, and when detecting a first identifier corresponding to the abnormal data, manually retransmitting the abnormal data.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the apparatus and the modules described above may refer to corresponding processes in the foregoing method embodiments, and are not described herein again.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. The electronic equipment is used for data storage. As shown in fig. 4, the electronic apparatus 4 of this embodiment includes: a memory 410, a processor 420, and a system bus 430, the memory 410 including an executable program 4101 stored thereon, it being understood by those skilled in the art that the electronic device configuration shown in fig. 4 does not constitute a limitation of electronic devices and may include more or fewer components than shown, or some components in combination, or a different arrangement of components.
The following describes each component of the electronic device in detail with reference to fig. 4:
the memory 410 may be used to store software programs and modules, and the processor 420 executes various functional applications and data processing of the electronic device by operating the software programs and modules stored in the memory 410. The memory 410 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as cache data) created according to the use of the electronic device, and the like. Further, the memory 410 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
The executable program 4101 of the network request method is contained on the memory 410, the executable program 4101 may be divided into one or more modules/units, the one or more modules/units are stored in the memory 410 and executed by the processor 420 to implement the driving data consistency check and the like, and the one or more modules/units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used for describing the execution process of the computer program 4101 in the electronic device 4. For example, the computer program 4101 may be divided into an upload module, a data transmission module, a generation module, and a comparison and retransmission module.
The processor 420 is a control center of the electronic device, connects various parts of the entire electronic device using various interfaces and lines, performs various functions of the electronic device and processes data by operating or executing software programs and/or modules stored in the memory 410 and calling data stored in the memory 410, thereby performing overall status monitoring of the electronic device. Alternatively, processor 420 may include one or more processing units; preferably, the processor 420 may integrate an application processor, which mainly handles operating systems, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into processor 420.
The system bus 430 is used to connect functional units inside the computer, and can transmit data information, address information, and control information, and may be, for example, a PCI bus, an ISA bus, a VESA bus, etc. The instructions of the processor 420 are transmitted to the memory 410 through the bus, the memory 410 feeds data back to the processor 420, and the system bus 430 is responsible for data and instruction interaction between the processor 420 and the memory 410. Of course, the system bus 430 may also access other devices such as network interfaces, display devices, and the like.
In this embodiment of the present invention, the executable program executed by the process 420 included in the electronic device includes:
generating a first json file based on local data pre-uploaded by a vehicle end, and uploading the first json file to Snowball equipment;
after the data in the Snowball equipment is sent to the data center, the data center transmits the data in the Snowball equipment to a storage bucket;
acquiring data in a storage bucket by calling a predetermined Python tool kit, and generating a second json file corresponding to the storage bucket data;
and comparing the first json file with the second json file, judging whether the file data are consistent, generating a third json file based on the comparison result, marking whether the data comparison is normal in the third json file, and retransmitting abnormal data by comparison.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A method of storing data, comprising:
generating a first json file based on local data pre-uploaded by a vehicle end, and uploading the first json file to Snowball equipment;
after the data in the Snowball equipment is sent to the data center, the data center transmits the data in the Snowball equipment to a storage bucket;
acquiring data in a storage bucket by calling a predetermined Python tool kit, and generating a second json file corresponding to the storage bucket data;
and comparing the first json file with the second json file, judging whether the file data are consistent, generating a third json file based on the comparison result, marking whether the data comparison is normal in the third json file, and retransmitting abnormal data by comparison.
2. The method of claim 1, wherein the first json file contains at least a file name and a data size of each piece of local data.
3. The method of claim 1, wherein the data center transmitting data in the Snowball device to the bucket comprises:
and executing a Snowball command for calling the AWSSCLL, and acquiring the data storage state through a cloud service interface.
4. The method of claim 1, wherein the obtaining data in the bucket by calling the predetermined Python toolkit is specifically:
and calling the boto3 toolkit in the Python to acquire the data in the cloud storage bucket.
5. The method of claim 1, wherein the third json file is marked whether the data comparison is normal, and wherein the retransmission of abnormal data by comparison comprises:
adding a first identifier for the abnormal comparison data in the third json file, and marking the normal comparison data as a second identifier;
and acquiring a third json file, and when detecting a first identifier corresponding to the abnormal data, manually retransmitting the abnormal data.
6. A data storage system, comprising:
the uploading module is used for generating a first json file based on local data pre-uploaded by the vehicle end and uploading the first json file to the Snowball equipment;
the data transmission module is used for transmitting the data in the Snowball equipment to the data center, and then the data center transmits the data in the Snowball equipment to the storage bucket;
the generating module is used for acquiring data in the storage bucket by calling a preset Python tool package and generating a second json file corresponding to the data of the storage bucket;
and the comparison retransmission module is used for comparing the first json file with the second json file, judging whether the file data are consistent, generating a third json file based on the comparison result, marking whether the data comparison is normal in the third json file, and retransmitting abnormal data by comparison.
7. The system of claim 6, wherein the first json file contains at least a file name and a data size of each piece of local data.
8. The system of claim 6, wherein the data center storing data in the Snowball device in buckets comprises:
and executing a Snowball command for calling the AWSSCLL, and acquiring the data storage state through a cloud service interface.
9. A terminal device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, characterized in that said processor implements the steps of the data storage method according to any one of claims 1 to 5 when executing said computer program.
10. A computer-readable storage medium, in which a computer program is stored, which, when executed, carries out the steps of the data storage method according to any one of claims 1 to 5.
CN202110862681.2A 2021-07-29 2021-07-29 Data storage method, system, electronic equipment and storage medium Pending CN113742301A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110862681.2A CN113742301A (en) 2021-07-29 2021-07-29 Data storage method, system, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110862681.2A CN113742301A (en) 2021-07-29 2021-07-29 Data storage method, system, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN113742301A true CN113742301A (en) 2021-12-03

Family

ID=78729393

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110862681.2A Pending CN113742301A (en) 2021-07-29 2021-07-29 Data storage method, system, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN113742301A (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180285369A1 (en) * 2017-03-29 2018-10-04 Amazon Technologies, Inc. Manifest generation for data transfers
CN109636948A (en) * 2019-01-31 2019-04-16 上海易点时空网络有限公司 Data processing method and device based on onboard system
CN109788039A (en) * 2018-12-26 2019-05-21 合肥凯捷技术有限公司 A kind of interactive system for vehicle-mounted voice storage method
CN110147470A (en) * 2019-05-22 2019-08-20 武汉市公安局 Across the computer room comparing system and method for one kind
CN112381891A (en) * 2020-11-30 2021-02-19 安徽江淮汽车集团股份有限公司 Method, device and equipment for acquiring automobile AVM calibration data and storage medium
CN112988499A (en) * 2019-12-16 2021-06-18 北京百度网讯科技有限公司 Data processing method and device, electronic equipment and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180285369A1 (en) * 2017-03-29 2018-10-04 Amazon Technologies, Inc. Manifest generation for data transfers
CN109788039A (en) * 2018-12-26 2019-05-21 合肥凯捷技术有限公司 A kind of interactive system for vehicle-mounted voice storage method
CN109636948A (en) * 2019-01-31 2019-04-16 上海易点时空网络有限公司 Data processing method and device based on onboard system
CN110147470A (en) * 2019-05-22 2019-08-20 武汉市公安局 Across the computer room comparing system and method for one kind
CN112988499A (en) * 2019-12-16 2021-06-18 北京百度网讯科技有限公司 Data processing method and device, electronic equipment and storage medium
CN112381891A (en) * 2020-11-30 2021-02-19 安徽江淮汽车集团股份有限公司 Method, device and equipment for acquiring automobile AVM calibration data and storage medium

Similar Documents

Publication Publication Date Title
US7849210B2 (en) Optimizing the responsiveness and throughput of a system performing packetized data transfers
CN102880532B (en) Cloud technology-based test system and method
CN103294580B (en) Real-time performance is monitored
CN109117158A (en) Firmware update and device
CN110297608B (en) Data processing method, device and storage medium
CN112272243B (en) Address allocation method
CN110460653A (en) The method and device of automatic driving vehicle data transmission
CN108647131B (en) Output system of running log
CN112433920A (en) Buried point data reporting method, device, equipment and medium
CN109408309A (en) The test method and device of multiple terminals
CN116860391A (en) GPU computing power resource scheduling method, device, equipment and medium
CN108234304A (en) Data transmission method, apparatus and system
CN102841825A (en) Drive backup method and device and drive restoration method and device
CN113742301A (en) Data storage method, system, electronic equipment and storage medium
CN109062580A (en) A kind of virtualized environment dispositions method and deployment device
CN115984022B (en) Unified account checking method and device for distributed payment system
CN108881493A (en) A kind of Task Progress information-pushing method, device and equipment
US20190109886A1 (en) Selected data exchange
JP5162686B2 (en) Method for retrieving object from DM client and device management system related thereto
CN109960657A (en) A kind of test environment dispositions method and relevant apparatus
CN111352781B (en) Message automatic comparison correctness checking method and device for retransmission component module level verification
WO2021073413A1 (en) Method and apparatus for sending system performance parameters, management device, and storage medium
CN111131369B (en) APP use condition transmission method and device, electronic equipment and storage medium
CN113852610A (en) Message processing method and device, computer equipment and storage medium
CN113434281A (en) Equipment scheduling method and cloud platform

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination