CN113742301A - Data storage method, system, electronic equipment and storage medium - Google Patents
Data storage method, system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113742301A CN113742301A CN202110862681.2A CN202110862681A CN113742301A CN 113742301 A CN113742301 A CN 113742301A CN 202110862681 A CN202110862681 A CN 202110862681A CN 113742301 A CN113742301 A CN 113742301A
- Authority
- CN
- China
- Prior art keywords
- data
- json file
- snowball
- comparison
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000013500 data storage Methods 0.000 title claims abstract description 26
- 235000019013 Viburnum opulus Nutrition 0.000 claims abstract description 39
- 244000071378 Viburnum opulus Species 0.000 claims abstract description 39
- 230000002159 abnormal effect Effects 0.000 claims abstract description 18
- 230000005540 biological transmission Effects 0.000 claims abstract description 10
- 238000004590 computer program Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 description 12
- 230000006870 function Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Human Computer Interaction (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a data storage method, a system, an electronic device and a storage medium, wherein the method comprises the following steps: generating a first json file based on local data pre-uploaded by a vehicle end, and uploading the first json file to Snowball equipment; after the data in the Snowball equipment is sent to the data center, the data center transmits the data in the Snowball equipment to a storage bucket; acquiring data in a storage bucket by calling a predetermined Python tool kit, and generating a second json file corresponding to the storage bucket data; and comparing the first json file with the second json file, judging whether the file data are consistent, generating a third json file based on the comparison result, marking whether the data comparison is normal in the third json file, and retransmitting abnormal data by comparison. Therefore, the vehicle data transmission and storage efficiency can be improved, and the consistency of the stored data can be effectively guaranteed.
Description
Technical Field
The invention belongs to the field of data processing, and particularly relates to a data storage method, a data storage system, electronic equipment and a storage medium.
Background
In the process of driving of a general vehicle, an automatic driving system needs to collect driving data of the vehicle in real time for storage, analysis, backup and the like. In the data storage process, the amount of data to be transmitted and stored is large, and the conventional hard disk storage is difficult to meet the requirement of transmission speed, so that the storage efficiency is low.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data storage method, a data storage system, an electronic device, and a storage medium, which are used to solve the problem of low storage efficiency of an existing hard disk.
In a first aspect of the embodiments of the present invention, a data storage method is provided, including:
generating a first json file based on local data pre-uploaded by a vehicle end, and uploading the first json file to Snowball equipment;
after the data in the Snowball equipment is sent to the data center, the data center transmits the data in the Snowball equipment to a storage bucket;
acquiring data in a storage bucket by calling a predetermined Python tool kit, and generating a second json file corresponding to the storage bucket data;
and comparing the first json file with the second json file, judging whether the file data are consistent, generating a third json file based on the comparison result, marking whether the data comparison is normal in the third json file, and retransmitting abnormal data by comparison.
In a second aspect of embodiments of the present invention, there is provided a data storage system, including:
the uploading module is used for generating a first json file based on local data pre-uploaded by the vehicle end and uploading the first json file to the Snowball equipment;
the data transmission module is used for transmitting the data in the Snowball equipment to the data center, and then the data center transmits the data in the Snowball equipment to the storage bucket;
the generating module is used for acquiring data in the storage bucket by calling a preset Python tool package and generating a second json file corresponding to the data of the storage bucket;
and the comparison retransmission module is used for comparing the first json file with the second json file, judging whether the file data are consistent, generating a third json file based on the comparison result, marking whether the data comparison is normal in the third json file, and retransmitting abnormal data by comparison.
In a third aspect of the embodiments of the present invention, there is provided an apparatus, including a memory, a processor, and a computer program stored in the memory and executable by the processor, where the processor executes the computer program to implement the steps of the method according to the first aspect of the embodiments of the present invention.
In a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, which stores a computer program, which when executed by a processor implements the steps of the method provided by the first aspect of the embodiments of the present invention.
In the embodiment of the invention, the local vehicle data are uploaded to the cloud for storage, so that the data transmission and storage speed can be improved, the TB-level data storage requirements can be met, and meanwhile, the json file comparison is carried out on the uploaded data, so that the data consistency can be ensured, and the data storage is accurate and efficient.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a schematic flowchart of a data storage method according to an embodiment of the present invention;
fig. 2 is another schematic flow chart of a data storage method according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a data storage system according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the embodiments described below are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be understood that the terms "comprises" and "comprising," when used in this specification or claims and in the accompanying drawings, are intended to cover a non-exclusive inclusion, such that a process, method or system, or apparatus that comprises a list of steps or elements is not limited to the listed steps or elements. In addition, "first" and "second" are used to distinguish different objects, and are not used to describe a specific order.
Referring to fig. 1, a flow chart of a data storage method according to an embodiment of the present invention includes:
s101, generating a first json file based on local data pre-uploaded by a vehicle end, and uploading the first json file to Snowball equipment;
before uploading driving data collected by a vehicle to a cloud end, a json file needs to be generated according to collected local data. The json file is a lightweight data exchange format, can represent or store data by adopting a text format, and can improve the data transmission efficiency by transmitting local data by adopting the json format.
The Snowball equipment is data storage intermediate equipment provided by the Amazon cloud service, can transfer local mass data into the Snowball, and transmits the mass data to the cloud service center through the Snowball.
The first json file at least contains the file name and the size of each piece of local data.
S102, after the data in the Snowball equipment are sent to a data center, the data center transmits the data in the Snowball equipment to a storage bucket;
and sending the Snowball equipment to an Amazon network service (AWS) center, and transmitting the data in the Snowball to an AWS cloud storage bucket by the data center. The Bucket (Bucket) is a carrier of the object and used for storing object data, and a user can create the Bucket and upload the object data according to an API provided by the Amazon cloud service.
Optionally, as shown in fig. 2, a Snowball command calling the AWSSCLL is executed, and the data storage state is acquired through the cloud service interface. And running a snowball interface program of the AWSCLL, acquiring the progress state of the snowball task in real time, and if the progress state is the completed state, indicating that the snowball operation is completed, namely the data transmission task on the snowball equipment is completed.
Wherein, awscll (amazon Web service command Line interface) is a command Line tool for managing AWS (amazon Web service) service, and uses corresponding service command to operate and access AWS service.
S103, acquiring data in a storage bucket by calling a preset Python tool kit, and generating a second json file corresponding to the storage bucket data;
and calling a boto3 toolkit in Python to read the cloud storage bucket data. For data stored in the AWS cloud bucket, the corresponding cloud data is obtained through the bato3 software development kit.
For vehicle end data stored in the cloud storage barrel, corresponding stored data are obtained through a reading command of the tool kit to generate a json file, and data consistency comparison is facilitated.
S104, comparing the first json file with the second json file, judging whether file data are consistent, generating a third json file based on a comparison result, marking whether data comparison is normal in the third json file, and retransmitting abnormal data through comparison.
The first json file is a json file generated according to local data, the second json file is a json file generated according to cloud storage data, the first json file and the second json file are compared, and whether all data of the first json file exist in the second json file or not and the sizes of the data are consistent are judged. Based on the comparison results, a corresponding json file may be generated.
Adding a first identifier for the abnormal comparison data in the third json file, and marking the normal comparison data as a second identifier; and acquiring a third json file, and when detecting a first identifier corresponding to the abnormal data, manually retransmitting the abnormal data.
In this embodiment, upload the data of gathering in the real vehicle road test and store and handle in the high in the clouds, promoted data transmission and processing speed, reduced the hard disk and used and the maintenance cost, conveniently carry out the efficient to large batch TB level data and handle.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
Fig. 3 is a schematic structural diagram of a data storage system according to an embodiment of the present invention, where the system includes:
the uploading module 310 is configured to generate a first json file based on local data pre-uploaded by a vehicle end, and upload the first json file to Snowball equipment;
the first json file at least contains the file name and the size of each piece of local data.
The data transmission module 320 is configured to transmit data in the Snowball equipment to the data center, and then the data center transmits the data in the Snowball equipment to the storage bucket;
optionally, a Snowball command for calling the AWSSCLL is executed, and the data storage state is acquired through the cloud service interface.
The generating module 330 is configured to obtain data in a bucket by calling a predetermined Python toolkit, and generate a second json file corresponding to the bucket data;
and acquiring data in the cloud storage bucket by calling the boto3 tool package in the Python.
And the comparison retransmission module 340 is configured to compare the first json file with the second json file, determine whether file data are consistent, generate a third json file based on the comparison result, mark whether data comparison is normal in the third json file, and retransmit abnormal data by comparison.
Specifically, a first identifier is added to the abnormal comparison data in the third json file, and the normal comparison data is marked as a second identifier; and acquiring a third json file, and when detecting a first identifier corresponding to the abnormal data, manually retransmitting the abnormal data.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the apparatus and the modules described above may refer to corresponding processes in the foregoing method embodiments, and are not described herein again.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. The electronic equipment is used for data storage. As shown in fig. 4, the electronic apparatus 4 of this embodiment includes: a memory 410, a processor 420, and a system bus 430, the memory 410 including an executable program 4101 stored thereon, it being understood by those skilled in the art that the electronic device configuration shown in fig. 4 does not constitute a limitation of electronic devices and may include more or fewer components than shown, or some components in combination, or a different arrangement of components.
The following describes each component of the electronic device in detail with reference to fig. 4:
the memory 410 may be used to store software programs and modules, and the processor 420 executes various functional applications and data processing of the electronic device by operating the software programs and modules stored in the memory 410. The memory 410 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as cache data) created according to the use of the electronic device, and the like. Further, the memory 410 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
The executable program 4101 of the network request method is contained on the memory 410, the executable program 4101 may be divided into one or more modules/units, the one or more modules/units are stored in the memory 410 and executed by the processor 420 to implement the driving data consistency check and the like, and the one or more modules/units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used for describing the execution process of the computer program 4101 in the electronic device 4. For example, the computer program 4101 may be divided into an upload module, a data transmission module, a generation module, and a comparison and retransmission module.
The processor 420 is a control center of the electronic device, connects various parts of the entire electronic device using various interfaces and lines, performs various functions of the electronic device and processes data by operating or executing software programs and/or modules stored in the memory 410 and calling data stored in the memory 410, thereby performing overall status monitoring of the electronic device. Alternatively, processor 420 may include one or more processing units; preferably, the processor 420 may integrate an application processor, which mainly handles operating systems, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into processor 420.
The system bus 430 is used to connect functional units inside the computer, and can transmit data information, address information, and control information, and may be, for example, a PCI bus, an ISA bus, a VESA bus, etc. The instructions of the processor 420 are transmitted to the memory 410 through the bus, the memory 410 feeds data back to the processor 420, and the system bus 430 is responsible for data and instruction interaction between the processor 420 and the memory 410. Of course, the system bus 430 may also access other devices such as network interfaces, display devices, and the like.
In this embodiment of the present invention, the executable program executed by the process 420 included in the electronic device includes:
generating a first json file based on local data pre-uploaded by a vehicle end, and uploading the first json file to Snowball equipment;
after the data in the Snowball equipment is sent to the data center, the data center transmits the data in the Snowball equipment to a storage bucket;
acquiring data in a storage bucket by calling a predetermined Python tool kit, and generating a second json file corresponding to the storage bucket data;
and comparing the first json file with the second json file, judging whether the file data are consistent, generating a third json file based on the comparison result, marking whether the data comparison is normal in the third json file, and retransmitting abnormal data by comparison.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method of storing data, comprising:
generating a first json file based on local data pre-uploaded by a vehicle end, and uploading the first json file to Snowball equipment;
after the data in the Snowball equipment is sent to the data center, the data center transmits the data in the Snowball equipment to a storage bucket;
acquiring data in a storage bucket by calling a predetermined Python tool kit, and generating a second json file corresponding to the storage bucket data;
and comparing the first json file with the second json file, judging whether the file data are consistent, generating a third json file based on the comparison result, marking whether the data comparison is normal in the third json file, and retransmitting abnormal data by comparison.
2. The method of claim 1, wherein the first json file contains at least a file name and a data size of each piece of local data.
3. The method of claim 1, wherein the data center transmitting data in the Snowball device to the bucket comprises:
and executing a Snowball command for calling the AWSSCLL, and acquiring the data storage state through a cloud service interface.
4. The method of claim 1, wherein the obtaining data in the bucket by calling the predetermined Python toolkit is specifically:
and calling the boto3 toolkit in the Python to acquire the data in the cloud storage bucket.
5. The method of claim 1, wherein the third json file is marked whether the data comparison is normal, and wherein the retransmission of abnormal data by comparison comprises:
adding a first identifier for the abnormal comparison data in the third json file, and marking the normal comparison data as a second identifier;
and acquiring a third json file, and when detecting a first identifier corresponding to the abnormal data, manually retransmitting the abnormal data.
6. A data storage system, comprising:
the uploading module is used for generating a first json file based on local data pre-uploaded by the vehicle end and uploading the first json file to the Snowball equipment;
the data transmission module is used for transmitting the data in the Snowball equipment to the data center, and then the data center transmits the data in the Snowball equipment to the storage bucket;
the generating module is used for acquiring data in the storage bucket by calling a preset Python tool package and generating a second json file corresponding to the data of the storage bucket;
and the comparison retransmission module is used for comparing the first json file with the second json file, judging whether the file data are consistent, generating a third json file based on the comparison result, marking whether the data comparison is normal in the third json file, and retransmitting abnormal data by comparison.
7. The system of claim 6, wherein the first json file contains at least a file name and a data size of each piece of local data.
8. The system of claim 6, wherein the data center storing data in the Snowball device in buckets comprises:
and executing a Snowball command for calling the AWSSCLL, and acquiring the data storage state through a cloud service interface.
9. A terminal device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, characterized in that said processor implements the steps of the data storage method according to any one of claims 1 to 5 when executing said computer program.
10. A computer-readable storage medium, in which a computer program is stored, which, when executed, carries out the steps of the data storage method according to any one of claims 1 to 5.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110862681.2A CN113742301A (en) | 2021-07-29 | 2021-07-29 | Data storage method, system, electronic equipment and storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110862681.2A CN113742301A (en) | 2021-07-29 | 2021-07-29 | Data storage method, system, electronic equipment and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
CN113742301A true CN113742301A (en) | 2021-12-03 |
Family
ID=78729393
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110862681.2A Pending CN113742301A (en) | 2021-07-29 | 2021-07-29 | Data storage method, system, electronic equipment and storage medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113742301A (en) |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180285369A1 (en) * | 2017-03-29 | 2018-10-04 | Amazon Technologies, Inc. | Manifest generation for data transfers |
CN109636948A (en) * | 2019-01-31 | 2019-04-16 | 上海易点时空网络有限公司 | Data processing method and device based on onboard system |
CN109788039A (en) * | 2018-12-26 | 2019-05-21 | 合肥凯捷技术有限公司 | A kind of interactive system for vehicle-mounted voice storage method |
CN110147470A (en) * | 2019-05-22 | 2019-08-20 | 武汉市公安局 | Across the computer room comparing system and method for one kind |
CN112381891A (en) * | 2020-11-30 | 2021-02-19 | 安徽江淮汽车集团股份有限公司 | Method, device and equipment for acquiring automobile AVM calibration data and storage medium |
CN112988499A (en) * | 2019-12-16 | 2021-06-18 | 北京百度网讯科技有限公司 | Data processing method and device, electronic equipment and storage medium |
-
2021
- 2021-07-29 CN CN202110862681.2A patent/CN113742301A/en active Pending
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180285369A1 (en) * | 2017-03-29 | 2018-10-04 | Amazon Technologies, Inc. | Manifest generation for data transfers |
CN109788039A (en) * | 2018-12-26 | 2019-05-21 | 合肥凯捷技术有限公司 | A kind of interactive system for vehicle-mounted voice storage method |
CN109636948A (en) * | 2019-01-31 | 2019-04-16 | 上海易点时空网络有限公司 | Data processing method and device based on onboard system |
CN110147470A (en) * | 2019-05-22 | 2019-08-20 | 武汉市公安局 | Across the computer room comparing system and method for one kind |
CN112988499A (en) * | 2019-12-16 | 2021-06-18 | 北京百度网讯科技有限公司 | Data processing method and device, electronic equipment and storage medium |
CN112381891A (en) * | 2020-11-30 | 2021-02-19 | 安徽江淮汽车集团股份有限公司 | Method, device and equipment for acquiring automobile AVM calibration data and storage medium |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7849210B2 (en) | Optimizing the responsiveness and throughput of a system performing packetized data transfers | |
CN102880532B (en) | Cloud technology-based test system and method | |
CN103294580B (en) | Real-time performance is monitored | |
CN109117158A (en) | Firmware update and device | |
CN110297608B (en) | Data processing method, device and storage medium | |
CN112272243B (en) | Address allocation method | |
CN110460653A (en) | The method and device of automatic driving vehicle data transmission | |
CN108647131B (en) | Output system of running log | |
CN112433920A (en) | Buried point data reporting method, device, equipment and medium | |
CN109408309A (en) | The test method and device of multiple terminals | |
CN116860391A (en) | GPU computing power resource scheduling method, device, equipment and medium | |
CN108234304A (en) | Data transmission method, apparatus and system | |
CN102841825A (en) | Drive backup method and device and drive restoration method and device | |
CN113742301A (en) | Data storage method, system, electronic equipment and storage medium | |
CN109062580A (en) | A kind of virtualized environment dispositions method and deployment device | |
CN115984022B (en) | Unified account checking method and device for distributed payment system | |
CN108881493A (en) | A kind of Task Progress information-pushing method, device and equipment | |
US20190109886A1 (en) | Selected data exchange | |
JP5162686B2 (en) | Method for retrieving object from DM client and device management system related thereto | |
CN109960657A (en) | A kind of test environment dispositions method and relevant apparatus | |
CN111352781B (en) | Message automatic comparison correctness checking method and device for retransmission component module level verification | |
WO2021073413A1 (en) | Method and apparatus for sending system performance parameters, management device, and storage medium | |
CN111131369B (en) | APP use condition transmission method and device, electronic equipment and storage medium | |
CN113852610A (en) | Message processing method and device, computer equipment and storage medium | |
CN113434281A (en) | Equipment scheduling method and cloud platform |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |