CN112001160B - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112001160B CN112001160B CN202010879229.2A CN202010879229A CN112001160B CN 112001160 B CN112001160 B CN 112001160B CN 202010879229 A CN202010879229 A CN 202010879229A CN 112001160 B CN112001160 B CN 112001160B
- Authority
- CN
- China
- Prior art keywords
- data
- slicing
- fragment
- value
- network order
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/186—Templates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/177—Editing, e.g. inserting or deleting of tables; using ruled lines
- G06F40/18—Editing, e.g. inserting or deleting of tables; using ruled lines of spreadsheets
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of data processing, and discloses a data processing method, a device, equipment and a storage medium, which are used for reducing the pressure of data to be processed on a database, wherein the method comprises the following steps: acquiring network order data; pre-slicing the network order data according to the dynamic first slicing value to generate a first slicing data set with uniform slicing; fragmenting the first fragment data in the first fragment data set according to a preset second fragment value to obtain a second fragment data set corresponding to the first fragment data; importing all the second fragment data sets into a template data table to generate a fragment data table; and loading the fragment data table into a processing platform, and respectively executing the instruction of each fragment data. By the method, updating operation caused by the segmentation step pair table can be reduced, the flow is simplified, database interaction is reduced, database resources are released, and the pressure of the database is reduced. In addition, the invention also relates to a blockchain technology, and data subjected to pre-slicing can be stored in the blockchain.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
The economy is rapidly developed, a large amount of commodity is continuously emerging, the commodity is as large as petroleum, brass, steel and aluminum, the commodity is as small as clothing, hats and shoes, and more people can trade. Meanwhile, the third information revolution in the world brings huge productivity and efficiency, and the trade of various articles for daily use and bulk goods on the network is gradually becoming a common trade mode for people. Network transactions, which have not been imaginable in the past, have now become a normative state of life for everyone. And the revolutionary pressure is continuously going to the industries of education, medical treatment, culture and transportation.
When the society is in the electronic trade age, the electronic trade platform is provided with a great opportunity and a challenge of having the capability of completing high concurrent trade. In the face of concurrent transactions, the processing speed of a single thread of a server is constant, and in order to improve efficiency, several parallel processes must be processed simultaneously, and when a database processes data, the data structure of the processed data is complex, the data parsing difficulty is high, and the performance is reduced.
Disclosure of Invention
The invention mainly aims to solve the technical problems of performance degradation caused by complex data structure and high data analysis difficulty of the processed data in the database when the database processes the data in the prior art.
The first aspect of the present invention provides a data processing method, including:
acquiring network order data;
pre-slicing the network order data according to a dynamic first slicing value to generate a first slicing data set, wherein the first slicing data comprises at least two pieces of first slicing data, and the data slices of each piece of first slicing data are equal in size;
performing slicing processing on each piece of first slicing data in the first slicing data set according to a preset second slicing value to obtain a corresponding second slicing data set;
importing all the second fragment data sets into a template data table to generate a fragment data table;
and loading the fragment data table into a processing platform, and executing the instruction of each fragment data in turn.
Optionally, in a first implementation manner of the first aspect of the present invention, after the acquiring network order data, the method further includes:
reading the data type of the network order data, and judging whether the data type is a digital type or not;
if the data type is a digital type, acquiring the dynamic first fragment value;
and if the data type is not the digital type, converting the network order data into the digital type and sending a confirmation mail to a control server.
Optionally, in a second implementation manner of the first aspect of the present invention, the obtaining the dynamic first slice value includes:
analyzing and obtaining the data volume of the network order data;
obtaining the number of dynamic fragments according to the data volume and a preset dynamic corresponding relation;
and obtaining a dynamic first fragment value corresponding to the network order data according to the quotient value of the data quantity and the dynamic fragment quantity.
Optionally, in a third implementation manner of the first aspect of the present invention, the performing, according to a preset second slicing value, slicing processing on each piece of first slicing data in the first slicing data set, to obtain a corresponding second slicing data set includes:
judging whether the dynamic first fragment value is larger than the second fragment value or not;
if the dynamic first fragment value is larger than the second fragment value, importing the second fragment data into a template data table to generate a fragment data table;
and if the dynamic first slicing value is not greater than the second slicing value, slicing the first slicing data according to the dynamic first slicing value to obtain a second slicing data set of a digital type.
Optionally, in a fourth implementation manner of the first aspect of the present invention, the importing all the second sliced data sets into a template data table, and generating the sliced data table includes:
Reading the form type of the template data table, and judging whether the form type is in a system processing type range;
if the form type is within the system processing type range, reading a template data table to sequentially write second fragment data in a second fragment data set corresponding to the first fragment data;
if the form type is not in the system processing type range, judging whether the form type belongs to the form type capable of being automatically corrected;
if the form type belongs to the form type capable of being automatically corrected, automatically correcting the form type, and writing second fragment data in the second fragment data set corresponding to the first fragment data into a corrected template data table.
Optionally, in a fifth implementation manner of the first aspect of the present invention, after the importing all the second slice data sets into the template data table, generating the slice data table further includes:
judging whether the data quantity in the fragmentation data table exceeds a read data quantity threshold value or not;
if the data quantity threshold value is exceeded, the sliced data list is split evenly according to the first sliced data until the data quantity of each split data list is smaller than the data quantity threshold value;
And if the read data quantity threshold is not exceeded, sending the fragment data table to a cloud database for backup.
Optionally, in a sixth implementation manner of the first aspect of the present invention, after the slicing the first sliced data in the first sliced data set according to the preset second sliced value, obtaining a second sliced data set corresponding to the first sliced data further includes:
scanning all second fragment data in the second fragment data set, and judging whether fragment error data exist or not;
if the piece-wise error data exists, acquiring repair piece-wise data from the network order data according to first piece-wise data corresponding to the piece-wise error data, and replacing the repair piece-wise data with the piece-wise error data;
and if the data errors occur in the written template data table, recovering the data from the locked backup.
A second aspect of the present invention provides a data processing apparatus comprising:
the acquisition module is used for acquiring network order data;
the first slicing module is used for pre-slicing the network order data according to dynamic first slicing values to generate a first slicing data set, wherein the first slicing data comprises at least two pieces of first slicing data, and the data slices of each piece of first slicing data are equal in size;
The second slicing module is used for carrying out slicing processing on each piece of first slicing data in the first slicing data set according to a preset second slicing value to obtain a corresponding second slicing data set;
the writing module is used for importing all the second fragment data sets into a template data table to generate a fragment data table;
and the execution module is used for loading the fragment data table into the processing platform and executing the instruction of each fragment data in turn.
Optionally, in a first implementation manner of the second aspect of the present invention, the data processing apparatus further includes a type judgment module, where the type judgment module includes:
the reading unit is used for reading the data type of the network order data and judging whether the data type is a digital type or not;
the score obtaining unit is used for obtaining the dynamic first fragment value when the data type is a digital type;
and the conversion unit is used for converting the network order data into a digital type and sending a confirmation mail to the control server when the data type is not the digital type.
Optionally, in a second implementation manner of the second aspect of the present invention, the score obtaining unit is specifically configured to:
Analyzing and obtaining the data volume of the network order data;
obtaining the number of dynamic fragments according to the data volume and a preset dynamic corresponding relation;
and obtaining a dynamic first fragment value corresponding to the network order data according to the quotient value of the data quantity and the dynamic fragment quantity.
Optionally, in a third implementation manner of the second aspect of the present invention, the second slicing module is specifically configured to:
judging whether the dynamic first fragment value is larger than the second fragment value or not;
if the dynamic first fragment value is larger than the second fragment value, importing the second fragment data into a template data table to generate a fragment data table;
and if the dynamic first slicing value is not greater than the second slicing value, slicing the first slicing data according to the dynamic first slicing value to obtain a second slicing data set of a digital type.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the writing module is specifically configured to:
reading the form type of the template data table, and judging whether the form type is in a system processing type range;
if the form type is within the system processing type range, reading a template data table to sequentially write second fragment data in a second fragment data set corresponding to the first fragment data;
If the form type is not in the system processing type range, judging whether the form type belongs to the form type capable of being automatically corrected;
if the form type belongs to the form type capable of being automatically corrected, automatically correcting the form type, and writing second fragment data in the second fragment data set corresponding to the first fragment data into a corrected template data table.
Optionally, in a fifth implementation manner of the second aspect of the present invention, the data processing apparatus further includes a threshold value judging module:
judging whether the data quantity in the fragmentation data table exceeds a read data quantity threshold value or not;
if the data quantity threshold value is exceeded, the sliced data list is split evenly according to the first sliced data until the data quantity of each split data list is smaller than the data quantity threshold value;
and if the read data quantity threshold is not exceeded, sending the fragment data table to a cloud database for backup.
Optionally, in a sixth implementation manner of the second aspect of the present invention, the data processing apparatus further includes a judgment error module, where the judgment error module is specifically configured to:
scanning all second fragment data in the second fragment data set, and judging whether fragment error data exist or not;
If the piece-wise error data exists, acquiring repair piece-wise data from the network order data according to first piece-wise data corresponding to the piece-wise error data, and replacing the repair piece-wise data with the piece-wise error data;
and if the data errors occur in the written template data table, recovering the data from the locked backup.
A third aspect of the present invention provides a data processing apparatus comprising: a memory and at least one processor, the memory having instructions stored therein, the memory and the at least one processor being interconnected by a line; the at least one processor invokes the instructions in the memory to cause the data processing apparatus to perform the data processing method described above.
A fourth aspect of the present invention provides a computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform the data processing method described above.
In the technical scheme of the invention, network order data are acquired; pre-slicing the network order data according to the dynamic first slicing value to generate a first slicing data set with uniform slicing; fragmenting the first fragment data in the first fragment data set according to a preset second fragment value to obtain a second fragment data set corresponding to the first fragment data; importing all the second fragment data sets into a template data table to generate a fragment data table; and loading the fragment data table into a processing platform, and respectively executing the instruction of each fragment data. The method for slicing the data can lead the slices of the data to be uniform, simplify the data structure through slicing processing, facilitate the reading and analysis of the data by a database or equipment, simultaneously finish slicing operation when generating the data to be inserted into a table, reduce the updating operation caused by slicing step to the table, simplify the flow, reduce the interaction of the database, release the database resources and lighten the pressure of the database. In addition, the invention also relates to a blockchain technology, and data subjected to pre-slicing can be stored in the blockchain.
Drawings
FIG. 1 is a schematic diagram of a first embodiment of a data processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a second embodiment of a data processing method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a third embodiment of a data processing method according to an embodiment of the present invention;
FIG. 4 is a diagram showing a fourth embodiment of a data processing method according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a fifth embodiment of a data processing method according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of an embodiment of a data processing apparatus according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of another embodiment of a data processing apparatus according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of an embodiment of a data processing apparatus in an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a data processing method, a device, equipment and a storage medium, wherein in the technical scheme of the invention, network order data are acquired; pre-slicing the network order data according to the dynamic first slicing value to generate a first slicing data set with uniform slicing; fragmenting the first fragment data in the first fragment data set according to a preset second fragment value to obtain a second fragment data set corresponding to the first fragment data; importing all the second fragment data sets into a template data table to generate a fragment data table; and loading the fragment data table into a processing platform, and respectively executing the instruction of each fragment data. The method for partitioning the data can lead the partitioned data to be uniform, and simultaneously completes the partitioning operation when generating the data insertion table, thereby reducing the updating operation caused by the partitioning step table, simplifying the flow, reducing the database interaction, releasing the database resources and relieving the pressure of the database. In addition, the invention also relates to a blockchain technology, and data subjected to pre-slicing can be stored in the blockchain.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.
For ease of understanding, a specific flow of an embodiment of the present invention is described below with reference to fig. 1, where a first embodiment of a data processing method in an embodiment of the present invention includes:
101. acquiring network order data;
it will be appreciated that the execution subject of the present invention may be a data processing apparatus, a terminal or a server, and is not limited herein. The embodiment of the invention is described by taking a server as an execution main body as an example.
It is emphasized that to ensure the privacy and security of the network order data, the network order data may be stored in a node of a blockchain.
In this embodiment, the network order data may be obtained directly from the mobile memory or directly from the mobile device by using a USB interface. In another embodiment, the data is obtained from a server of the network using a wireless communication network or a physical network cable.
102. Pre-slicing the network order data according to the dynamic first slicing value to generate a first slicing data set;
in this embodiment, before slicing, the dynamic first slicing value is determined according to a table or a function of a one-to-one correspondence between a preset slicing value and a data amount, for example, if the dynamic first slicing value is 500, and the data amount of the network order data is 10000, an average of 20 slicing data are generated during slicing, and the 20 slicing data together form the first slicing data set. In practical application, in a normal operation link, generally stored data can be gradually increased, only 10 fragments are needed at present, but after a certain time, 50 fragments are needed to support, capacity expansion is needed, in distributed deployment, pre-fragmentation is a very good method, 50 fragments are performed in advance, 10 fragments are performed in 50 fragments, and when capacity expansion is needed later, partial data in the data are transferred to other fragments. And the network order data is not necessarily completely cut every time, when the first slice value is 500, and the data amount of the network order data is 10300, an average of 21 slices of data is generated when slicing, but the last slice is 300 data amounts.
103. Performing slicing processing on each piece of first slicing data in the first slicing data set according to a preset second slicing value to obtain a corresponding second slicing data set;
in this embodiment, the second slice value is internally set, and is a value of a slice actually required, for example, 50, data of {500,5, 500, 500, 500, 500, 500, 500, 300} is sliced respectively in the first slice data set and then data of which each element of each set can be sliced according to data of 50, for example, 500 is divided into {50, 50, 50, 50, 50, 50, 50, 50, 50} 10 pieces of data in total, are sliced data, and data of 300 is divided into {50, 50, 50, 50, 50, 50}, and slices of each 50 data amount in the data are each numbered 1 to 50. Each first sliced data is then divided by the data of the preset second sliced value, resulting in a corresponding second sliced data set.
104. Importing all the second fragment data sets into a template data table to generate a fragment data table;
in this embodiment, all the second tile data {50, 50, 50, 50, 50, 50, 50, 50}, {50, 50, 50, 50, 50, 50, 50, 50, 50}, … …, {50, 50, 50, 50, 50, 50, 50, 50, 50} are written into the template data table, and then the template data table in which data is written is the tile data table, wherein the template data table is a data table set in advance for performing the writing of the tile data. In one embodiment, the xls format of the Excel table is used as the template data table, and the elements of each second piece of data are written in a row and marked with "#" as the end, according to the sequential writing.
105. And loading the fragment data table into a processing platform, and executing the instruction of each fragment data in turn.
In this embodiment, the network order data processing forms {1,1,2,3,6}, {1,8,2,6,6}, {1,1,4,3,6}, {1,1,4,3,6}, and {1,1,4,3,6}, so that the data slicing uses cpu to process {1,1,2,3,6}, {1,8,2,6,6}, {1,1,4,3,6}, {1,1,4,3,6}, and {1,1,4,3,6} each data content to process each data content, 5 data are compiled first, then the 5 data are processed simultaneously, and the final speed is increased by 5 times, and when the data is processed, the data slicing is not re-processed.
In this embodiment, the network order data is acquired; pre-slicing the network order data according to the dynamic first slicing value to generate a first slicing data set with uniform slicing; fragmenting the first fragment data in the first fragment data set according to a preset second fragment value to obtain a second fragment data set corresponding to the first fragment data; importing all the second fragment data sets into a template data table to generate a fragment data table; and loading the fragment data table into a processing platform, and respectively executing the instruction of each fragment data. The method for partitioning the data can lead the partitioned data to be uniform, and simultaneously completes the partitioning operation when generating the data insertion table, thereby reducing the updating operation caused by the partitioning step table, simplifying the flow, reducing the database interaction, releasing the database resources and relieving the pressure of the database.
Referring to fig. 2, a second embodiment of a data processing method according to an embodiment of the present invention includes:
201. acquiring network order data;
202. reading the data type of the network order data, and judging whether the data type is a digital type or not;
in this embodiment, the type of network order data is read and, if in text format, needs to be changed to the data type. In checking the data type, the type may be used to detect the data type, firstly, a character string is returned, and secondly, the character string includes the corresponding data type, for example: "number", "string", "spool", "undefined", "function", "object". In addition, the judgment can also be made using, for example, an object.
203. If the data type is a digital type, analyzing and obtaining the data volume of the network order data;
in this embodiment, when the data type is a digital type, it may be determined whether the network order data reaches the threshold. In order to save resources, data which do not need to be fragmented are not fragmented, for example, a basic line with 8000 data volume being data fragmentation is set, and only the basic line reaching the data volume can be fragmented. How to not reach the data slicing threshold, the slicing is not directly carried out, and the network order data is directly stored.
In this embodiment, the data size of the network order data is read, the data size may be 500M, then the system analyzes 500M for 524288000 bytes, and after converting the read byte size, it may be determined whether the byte size overflows.
204. Obtaining the number of dynamic fragments according to the data volume and a preset dynamic corresponding relation;
in this embodiment, a dynamic relation table is built in, the number of dynamic slices smaller than 1000 is 500, the number of dynamic slices larger than 1000 and smaller than 10000 is 1000, the number of dynamic slices larger than 10000 and smaller than 100000 is 5000, and the number of dynamic slices larger than 100000 and smaller than 1000000 is 50000. And the acquired network order data has the data quantity of 80000, and the dynamic fragment quantity is 5000.
205. Dynamic first fragment values corresponding to the network order data are obtained according to the quotient of the data quantity and the dynamic fragment quantity;
in this embodiment, the quotient is a value obtained by dividing the data amount by the dynamic slice number, for example, the data amount is 80000 bytes, and the dynamic slice number is 5000, and the quotient obtained by dividing 80000 bytes by 5000 is 16, so that the first slice value of slice is 16, that is, the network order data is equally divided into 16 slices. In another embodiment, the data volume of the network order data fluctuates to cause the dynamic first tile value to correspondingly change, a threshold is set when the data volume continuously fluctuates to increase, for example, 3 seconds, and if the data does not increase within 3 seconds, the calculation of the data volume is stopped and then the analysis of the data volume and the tile number and the dynamic first tile value is started.
206. If the data type is not the digital type, converting the network order data into the digital type, and sending a confirmation mail to the control server;
in this embodiment, the network order data is converted into a digital type, from a text type to a digital type, a string.format ("% 02d", num) function may be used to perform conversion, and when the data conversion is completed, a converted confirmed function piece may be sent to the control server, and if the function piece is confirmed, the next slicing step is continued.
207. Pre-slicing the network order data according to the dynamic first slicing value to generate a first slicing data set;
208. performing slicing processing on each piece of first slicing data in the first slicing data set according to a preset second slicing value to obtain a corresponding second slicing data set;
209. importing all the second fragment data sets into a template data table to generate a fragment data table;
210. and loading the fragment data table into a processing platform, and executing the instruction of each fragment data in turn.
Steps 201, 205-208 in this embodiment are similar to steps 101-105 in the first embodiment, and will not be described here again.
The embodiment adds the process of judging whether the data is of a digital type or not before the data slicing is carried out, converts the data which is not of the digital type into the digital type, stores the data in the digital type, saves the storage space, can improve the query positioning speed, reduces the updating operation caused by the slicing step pair table because the query number is faster than the query character string, simultaneously describes the process of obtaining the dynamic first slicing value in detail, obtains the data volume of the network order data through analysis, obtains the dynamic slicing number according to the data volume and the preset dynamic corresponding relation, obtains the first slicing value according to the quotient of the data volume and the dynamic slicing number, can pre-slice the network order data through the first slicing value, generates the first slicing data set with even slicing, can ensure that the slices of the data are even, simultaneously completes the slicing operation when the data is generated in the data insertion table, can reduce the updating operation caused by the slicing step pair table, simplify the database interaction, and lighten the pressure of the database.
Referring to fig. 3, a third embodiment of a data processing method according to an embodiment of the present invention includes:
301. acquiring network order data;
302. pre-slicing the network order data according to the dynamic first slicing value to generate a first slicing data set;
303. judging whether the dynamic first fragment value is larger than the second fragment value or not;
in this embodiment, if the dynamic first slice value is 80 average slices and the second slice value is 40 slices, it is indicated that the subdivision degree of the second slice value is smaller than that of the first slice value, which results in a reduction of the computation of the computer. If the subdivision degree is increased, huge data calculation power is brought, and the situation is processed. In the case of statistical complexity, the size of the quantized slice value may be used as the load pressure during the computation.
304. If the dynamic first fragment value is larger than the second fragment value, importing the second fragment data into a template data table to generate a fragment data table;
in this embodiment, if the dynamic first slice value is 50 and the second slice value is 40, it may be explained that the complexity is decreasing, and the data processing disaster is not caused, and the data is processed according to the second slice value, so as to generate the slice data table.
305. If the dynamic first slicing value is not greater than the second slicing value, slicing the first slicing data according to the dynamic first slicing value to obtain a second slicing data set of a digital type;
in this embodiment, the data complexity is increasing, the dynamic first fragment value is smaller than the second fragment value, the data complexity is uncontrollable, and the possible second fragment value may be 10000, which directly results in the direct crash of the down system, so that the situation needs to be avoided. Complexity is controllable, then the first sliced data is sliced using the dynamic first sliced value. In another embodiment, the threshold setting for the second sliced value may be 2 times the dynamic first sliced value, such that more than 2 times the dynamic first sliced value can be sliced using twice the value of the first sliced value. The product of the dynamic first slicing value and the parameter a can also be used as a slicing threshold, and the decision of the parameter a can be obtained by calculating the content of other algorithms.
306. Importing all the second fragment data sets into a template data table to generate a fragment data table;
307. and loading the fragment data table into a processing platform, and executing the instruction of each fragment data in turn.
The process of acquiring the second sliced data set is described in detail on the basis of the previous embodiment, and by judging whether the dynamic first sliced value is larger than the second sliced value or not, if the dynamic first sliced value is not larger than the second sliced value, slicing the first sliced data according to the dynamic first sliced value to obtain digital type second sliced data, and judging whether the second sliced data in the second sliced data set has sliced error data or not, errors in subsequent processes can be avoided, updating operations caused by slicing steps are reduced, the process is simplified, database interaction is reduced, database resources are released, and pressure of a database is relieved.
Referring to fig. 4, a fourth embodiment of a data processing method according to an embodiment of the present invention includes:
401. acquiring network order data;
402. pre-slicing the network order data according to the dynamic first slicing value to generate a first slicing data set;
403. performing slicing processing on each piece of first slicing data in the first slicing data set according to a preset second slicing value to obtain a corresponding second slicing data set;
404. reading the form type of the template data table, and judging whether the form type is in the range of the system processing type;
in this embodiment, the system supports Excel type tables for data operation, while non-Excel tables are not within the processing scope. In another embodiment, version 1.0 supports Excel all types of tables, while updated version 2.0 supports expanding txt, equp, word and so on, the system's processing scope expands and can continue to support new content.
405. If the form type is within the system processing type range, reading a template data table to sequentially write the second fragment data in the second fragment data set corresponding to the first fragment data into the form data table;
in this embodiment, if the table is within the supported range, for example, an Excel type table, then there is no abnormal situation, and the generated second fragment data is written into the data.
406. If the form type is not in the system processing type range, judging whether the form type belongs to the form type capable of being automatically corrected;
in this embodiment, the system may have an automatic correction function for a part of table types, and for a table type of the same major class, when the table type is not within the scope of the system processing type, the table type may be automatically corrected to a table type of the same major class within the scope of the system processing type for processing. If txt cannot be supported by the system operation, judging whether the data table type of the txt type can be repaired to be a type supported by the system, such as an Excel type. And if the data type can be repaired into the Excel type data type, performing the repair and then performing the writing again in the template data table.
407. If the form type belongs to the form type capable of being automatically corrected, automatically correcting the form type, and writing second fragment data in a second fragment data set corresponding to the first fragment data into a corrected template data table;
in this embodiment, if the data type is a table type that the jpg type cannot be corrected to be processable, a notification message needs to be sent to the management server, notifying that the management cannot be converted, and the management end can perform manual repair. In another embodiment, an update application is first performed to the cloud database, whether the version type is up to date is checked, and if the version still needs to be updated, the table is revised after the update is completed.
408. Judging whether the data quantity in the fragmentation data table exceeds a read data quantity threshold value or not;
in this embodiment, the fragmentation data table needs to determine the data amount in addition to the format requirement, and if the data amount exceeds the threshold, an unnecessary database load is brought and the range of the loop processing statement is exceeded, so that the fragmentation data table needs to be decomposed, and the number of times of the loop statement is ensured not to overflow.
409. If the data quantity threshold value is exceeded, the sliced data list is split evenly according to the first sliced data until the data quantity of each split data list is smaller than the data quantity threshold value;
in this embodiment, the data size has exceeded the identifiable threshold value, so that the data size of the sliced data table must be reduced, for example, the data size of the sliced data table is 500M, and the threshold value is 100M, so that the data of the sliced data table is divided into 250M and 250M, or exceeds the data size threshold value, and thus is reduced to 125M, 125M. At this time, if the threshold value is still exceeded, the data is continuously reduced to 75.5M, 75.5M and 75.5M, and the data amount is lower than 100M, and the decomposed number-divided data sheets are numbered as 1, 2, 3, 4, 5, 6, 7 and 8 and then stored.
410. If the read data quantity threshold is not exceeded, sending the fragment data table to a cloud database for backup;
in this embodiment, if the threshold is not exceeded, the shard data table is directly stored in the cloud database, so that the shard data table is convenient to use in next call. In another embodiment, the data line is used to transmit the fragmented data table to the storage device, and the storage device uses the mirror image to complete the data storage after the disaster recovery backup in different places.
411. And loading the fragment data table into a processing platform, and executing the instruction of each fragment data in turn.
The embodiment describes in detail the process of importing all the second sliced data sets into the template data table and generating the sliced data table based on the previous embodiment, and needs to read the table type of the template data table first, determine whether the table type is within the system processing type range, and write all the second sliced data in the second sliced data set corresponding to the first sliced data into the read template data table in turn, which is not within the system processing type range any more, but belongs to the automatically corrected table type range for automatic correction, and meanwhile, judge whether the data amount in the sliced data table exceeds the read data amount threshold value, and the data which does not reach the threshold value is not sliced, so that the resource can be saved.
Referring to fig. 5, a fifth embodiment of a data processing method according to an embodiment of the present invention includes:
501. acquiring network order data;
502. pre-slicing the network order data according to the dynamic first slicing value to generate a first slicing data set;
503. performing slicing processing on each piece of first slicing data in the first slicing data set according to a preset second slicing value to obtain a corresponding second slicing data set;
504. scanning the second fragment data in all the second fragment data sets, and judging whether fragment error data exist or not;
in the present embodiment, scanning the second piece of data finds that the character of the data of "first piece of data No. 15-second piece of data No. 25" is not within the specified character range, the data storage is considered to be erroneous. The storage error may be caused by a packet loss or a network instability. The data of the second fragment data number 25 can be directly replaced in the data discovery area, or the characters of the second fragment data number 25-16 can be continuously scanned to discover that the characters of the second fragment data number 16 exceed the range, and the characters of the second fragment data number 16 can be directly replaced.
505. If the fragmentation error data exists, acquiring repair fragmentation data from the network order data according to the first fragmentation data corresponding to the fragmentation error data, and replacing the repair fragmentation data with the fragmentation error data;
In this embodiment, when the situation that the data is lost or wrong during the processing occurs during the slicing process, the correct data needs to be found in the starting data set to replace, in this embodiment, the correct data corresponding to the slicing error data is obtained from the network order data, that is, the repair slicing data is obtained, the error data is replaced, repair of the slicing error data is completed, when the data exceeding the character range is found in the second slicing data number 25, the corresponding first slicing data of the second slicing data number 25 is queried, and then the position of the first slicing data in the network order data is queried. And then, original processing data corresponding to the second piece of data No. 25 is found from the network order data, and the piece of original data is intercepted and then copied into the second piece of data No. 25 to replace the original data.
506. If the data in the data table is not in the data in the template, the data is recovered from the locked backup;
in this embodiment, if the analysis finds that the sliced data is considered to have no error data, the sliced data is locked and other processing is performed based on the sliced data, and the sliced data table becomes the source data, and if the derived data is in error, the sliced data table can be used for recovery. The fragmented data table is stored in a built-in storage device, and then the data can be restored after a problem occurs in a later scan.
507. Importing all the second fragment data sets into a template data table to generate a fragment data table;
508. and loading the fragment data table into a processing platform, and respectively executing the instruction of each fragment data.
The embodiment adds a process of judging whether the error data exists in the written template data table on the basis of the previous embodiment, when the error data exists, acquires the repair fragment data from the network order data according to the first fragment data corresponding to the fragment error data, replaces the fragment error data with the repair fragment data, and when the error data does not exist, locks the obtained fragment data table in the built-in storage device, and if the data error occurs in the written template data table, recovers the data from the locked backup. By the method, data slicing can be performed, slices of data slicing can be uniform, slicing operation is completed when data insertion tables are generated, updating operation caused by slicing step pairs can be reduced, flow is simplified, database interaction is reduced, database resources are released, and pressure of a database is relieved.
The data processing method in the embodiment of the present invention is described above, and the data processing apparatus in the embodiment of the present invention is described below, referring to fig. 6, where an embodiment of the data processing apparatus in the embodiment of the present invention includes:
An acquisition module 601, configured to acquire network order data;
the first slicing module 602 is configured to perform pre-slicing processing on the network order data according to a dynamic first slicing value to generate a first slicing data set, where the first slicing data includes at least two pieces of first slicing data, and a data slice size of each piece of first slicing data is equal;
a second slicing module 603, configured to perform slicing processing on each piece of first sliced data in the first sliced data set according to a preset second sliced value, so as to obtain a corresponding second sliced data set;
a writing module 604, configured to import all the second fragment data sets into a template data table, and generate a fragment data table;
and the execution module 605 is used for loading the fragment data table into the processing platform and executing the instruction of each fragment data in turn.
It is emphasized that to ensure the privacy and security of the network order data, the network order data may be stored in a node of a blockchain.
In an embodiment of the present invention, the data processing apparatus runs the above data processing method, where the data processing method includes: acquiring network order data; pre-slicing the network order data according to the dynamic first slicing value to generate a first slicing data set with uniform slicing; fragmenting the first fragment data in the first fragment data set according to a preset second fragment value to obtain a second fragment data set corresponding to the first fragment data; importing all the second fragment data sets into a template data table to generate a fragment data table; and loading the fragment data table into a processing platform, and respectively executing the instruction of each fragment data. The method for partitioning the data can lead the partitioned data to be uniform, and simultaneously completes the partitioning operation when generating the data insertion table, thereby reducing the updating operation caused by the partitioning step table, simplifying the flow, reducing the database interaction, releasing the database resources and relieving the pressure of the database.
Referring to fig. 7, another embodiment of the data processing apparatus according to the present invention includes:
an acquisition module 601, configured to acquire network order data;
the first slicing module 602 is configured to perform pre-slicing processing on the network order data according to a dynamic first slicing value to generate a first slicing data set, where the first slicing data includes at least two pieces of first slicing data, and a data slice size of each piece of first slicing data is equal;
a second slicing module 603, configured to perform slicing processing on each piece of first sliced data in the first sliced data set according to a preset second sliced value, so as to obtain a corresponding second sliced data set;
a writing module 604, configured to import all the second fragment data sets into a template data table, and generate a fragment data table;
and the execution module 605 is used for loading the fragment data table into the processing platform and executing the instruction of each fragment data in turn.
Wherein the data processing apparatus further comprises a type judgment module 606, the type judgment module 606 comprises:
a reading unit 6061 for reading the data type of the network order data and judging whether the data type is a digital type;
A score acquisition unit 6062 for acquiring the dynamic first fragment value when the data type is a digital type;
a conversion unit 6063 for converting the network order data into a digital type when the data type is not a digital type, and transmitting a confirmation mail to the control server.
Optionally, the score obtaining unit 6062 is specifically configured to:
analyzing and obtaining the data volume of the network order data;
obtaining the number of dynamic fragments according to the data volume and a preset dynamic corresponding relation;
and obtaining a dynamic first fragment value corresponding to the network order data according to the quotient value of the data quantity and the dynamic fragment quantity.
Optionally, the second slicing module 603 is specifically configured to:
judging whether the dynamic first fragment value is larger than the second fragment value or not;
if the dynamic first fragment value is larger than the second fragment value, importing the second fragment data into a template data table to generate a fragment data table;
and if the dynamic first slicing value is not greater than the second slicing value, slicing the first slicing data according to the dynamic first slicing value to obtain a second slicing data set of a digital type.
Optionally, the writing module 604 is specifically configured to:
reading the form type of the template data table, and judging whether the form type is in a system processing type range;
if the form type is within the system processing type range, reading a template data table to sequentially write second fragment data in a second fragment data set corresponding to the first fragment data;
if the form type is not in the system processing type range, judging whether the form type belongs to the form type capable of being automatically corrected;
if the form type belongs to the form type capable of being automatically corrected, automatically correcting the form type, and writing second fragment data in the second fragment data set corresponding to the first fragment data into a corrected template data table.
Wherein the data processing apparatus further comprises a threshold value judging module 607:
judging whether the data quantity in the fragmentation data table exceeds a read data quantity threshold value or not;
if the data quantity threshold value is exceeded, the sliced data list is split evenly according to the first sliced data until the data quantity of each split data list is smaller than the data quantity threshold value;
And if the read data quantity threshold is not exceeded, sending the fragment data table to a cloud database for backup.
The data processing apparatus further includes a judgment error module 608, where the judgment error module 608 is specifically configured to:
scanning all second fragment data in the second fragment data set, and judging whether fragment error data exist or not;
if the piece-wise error data exists, acquiring repair piece-wise data from the network order data according to first piece-wise data corresponding to the piece-wise error data, and replacing the repair piece-wise data with the piece-wise error data;
and if the data errors occur in the written template data table, recovering the data from the locked backup.
The embodiment describes the specific functions of each module in detail based on the previous embodiment, the score obtaining module obtains the data volume of the network order data through analysis, obtains the dynamic slicing number according to the data volume and the preset dynamic corresponding relation, obtains the first slicing value according to the quotient of the data volume and the dynamic slicing number, and the type judging module judges whether the data type is a digital type or not by reading the data type of the network order data, and converts the data type which is not stored into a numerical type, and stores the data type into the numerical type, namely, the storage space is saved, the speed of inquiring and positioning can be improved, because the inquiring number is faster than the inquiring character string, the threshold judging module judges whether the data volume in the slicing data table exceeds the reading data volume threshold value, and does not perform slicing on the data which does not reach the data volume threshold value, resources can be saved, the slicing operation of the data can be completed through the original module and the new adding module, the slicing operation of the data can be uniform, the updating operation caused by the slicing step can be reduced, the simplification of the database, the interaction is reduced, the database is released, and the pressure resource of the database is relieved.
The data processing apparatus in the embodiment of the present invention is described in detail above in terms of modular functional entities in fig. 6 and 7, and the data processing device in the embodiment of the present invention is described in detail below in terms of hardware processing.
Fig. 8 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention, where the data processing apparatus 800 may have a relatively large difference due to different configurations or performances, and may include one or more processors (central processing units, CPU) 810 (e.g., one or more processors) and a memory 820, and one or more storage media 830 (e.g., one or more mass storage devices) storing application programs 833 or data 832. Wherein memory 820 and storage medium 830 can be transitory or persistent. The program stored on the storage medium 830 may include one or more modules (not shown), each of which may include a series of instruction operations in the data processing apparatus 800. Still further, the processor 810 may be arranged to communicate with the storage medium 830 and execute a series of instruction operations in the storage medium 830 on the data processing apparatus 800 to implement the steps of the data processing method provided in the above embodiments.
The data processing device 800 may also include one or more power supplies 840, one or more wired or wireless network interfaces 850, one or more input/output interfaces 860, and/or one or more operating systems 831, such as Windows Serve, mac OS X, unix, linux, freeBSD, and the like. It will be appreciated by persons skilled in the art that the data processing apparatus structure shown in fig. 8 is not limiting of the data processing apparatus provided herein, and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm and the like. The Blockchain (Blockchain), which is essentially a decentralised database, is a string of data blocks that are generated by cryptographic means in association, each data block containing a batch of information of network transactions for verifying the validity of the information (anti-counterfeiting) and generating the next block. The blockchain may include a blockchain underlying platform, a platform product services layer, an application services layer, and the like.
The present invention also provides a computer readable storage medium, which may be a non-volatile computer readable storage medium, or a volatile computer readable storage medium, having stored therein instructions which, when executed on a computer, cause the computer to perform the steps of the data processing method.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the system or apparatus and unit described above may refer to the corresponding process in the foregoing method embodiment, which is not repeated herein.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (5)
1. A network order data processing method, characterized in that the network order data processing method comprises:
acquiring network order data;
pre-slicing the network order data according to a dynamic first slicing value to generate a first slicing data set, wherein the first slicing data set comprises at least two pieces of first slicing data, and the data slices of each piece of first slicing data are equal in size;
performing slicing processing on each piece of first slicing data in the first slicing data set according to a preset second slicing value to obtain a corresponding second slicing data set;
importing all the second fragment data sets into a template data table to generate a fragment data table;
Loading the fragment data table into a processing platform, and executing the instruction of each fragment data in turn;
after the network order data is acquired, the pre-slicing processing is performed on the network order data according to the dynamic first slicing value, and before the first slicing data set is generated, the method further comprises:
reading the data type of the network order data, and judging whether the data type is a digital type or not;
if the data type is a digital type, acquiring the dynamic first fragment value;
if the data type is not the digital type, converting the network order data into the digital type, and sending a confirmation mail to a control server;
the obtaining the dynamic first fragment value includes:
analyzing and obtaining the data volume of the network order data;
obtaining the number of dynamic fragments according to the data volume and a preset dynamic corresponding relation;
obtaining a dynamic first fragment value corresponding to the network order data according to the quotient value of the data quantity and the dynamic fragment quantity;
the step of importing all the second fragment data sets into a template data table, and the step of generating the fragment data table comprises the following steps:
reading the form type of the template data table, and judging whether the form type is in a system processing type range;
If the form type is within the system processing type range, reading a template data table to sequentially write second fragment data in a second fragment data set corresponding to the first fragment data;
if the form type is not in the system processing type range, judging whether the form type belongs to the form type capable of being automatically corrected;
if the form type belongs to the form type capable of being automatically corrected, automatically correcting the form type, and writing second fragment data in a second fragment data set corresponding to the first fragment data into a corrected template data table;
after said importing all of said second shard data sets into the template data table, generating the shard data table, further comprising:
judging whether the data quantity in the fragmentation data table exceeds a read data quantity threshold value or not;
if the data quantity threshold value is exceeded, the sliced data list is split evenly according to the first sliced data until the data quantity of each split data list is smaller than the data quantity threshold value;
if the read data quantity threshold is not exceeded, the fragment data table is sent to a cloud database for backup;
after the slicing processing is performed on each piece of first slicing data in the first slicing data set according to the preset second slicing value to obtain the corresponding second slicing data set, the method further comprises:
Scanning all second fragment data in the second fragment data set, and judging whether fragment error data exist or not;
if the piece-wise error data exists, acquiring repair piece-wise data from the network order data according to first piece-wise data corresponding to the piece-wise error data, and replacing the repair piece-wise data with the piece-wise error data;
and if the data errors occur in the written template data table, recovering the data from the locked backup.
2. The network order data processing method according to claim 1, wherein the performing the slicing process on each piece of the first sliced data in the first sliced data set according to the preset second sliced value, to obtain the corresponding second sliced data set includes:
judging whether the dynamic first fragment value is larger than the second fragment value or not;
if the dynamic first fragment value is larger than the second fragment value, importing the second fragment data into a template data table to generate a fragment data table;
and if the dynamic first slicing value is not greater than the second slicing value, slicing the first slicing data according to the dynamic first slicing value to obtain a second slicing data set of a digital type.
3. A network order data processing apparatus for use in the network order data processing method of claim 1, wherein the network order data processing apparatus comprises:
the acquisition module is used for acquiring network order data;
the first slicing module is used for pre-slicing the network order data according to dynamic first slicing values to generate a first slicing data set, wherein the first slicing data set comprises at least two pieces of first slicing data, and the data slices of each piece of first slicing data are equal in size;
the second slicing module is used for carrying out slicing processing on each piece of first slicing data in the first slicing data set according to a preset second slicing value to obtain a corresponding second slicing data set;
the writing module is used for importing all the second fragment data sets into a template data table to generate a fragment data table;
and the execution module is used for loading the fragment data table into the processing platform and executing the instruction of each fragment data in turn.
4. A network order data processing device, the network order data processing device comprising: a memory and at least one processor, the memory having instructions stored therein, the memory and the at least one processor being interconnected by a line;
The at least one processor invoking the instructions in the memory to cause the network order data processing apparatus to perform the network order data processing method of any of claims 1-2.
5. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the network order data processing method according to any of claims 1-2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010879229.2A CN112001160B (en) | 2020-08-27 | 2020-08-27 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010879229.2A CN112001160B (en) | 2020-08-27 | 2020-08-27 | Data processing method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112001160A CN112001160A (en) | 2020-11-27 |
| CN112001160B true CN112001160B (en) | 2023-07-28 |
Family
ID=73471224
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010879229.2A Active CN112001160B (en) | 2020-08-27 | 2020-08-27 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112001160B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112750009A (en) * | 2021-01-05 | 2021-05-04 | 江苏壹集购贸易有限公司 | Intelligent hotel sale management system for theme banquet |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107967284A (en) * | 2016-10-20 | 2018-04-27 | 北京京东尚科信息技术有限公司 | Method and apparatus for storing, inquiring about sequence information |
| CN109189785A (en) * | 2018-08-10 | 2019-01-11 | 平安科技（深圳）有限公司 | Date storage method, device, computer equipment and storage medium |
| CN109669929A (en) * | 2018-12-14 | 2019-04-23 | 江苏瑞中数据股份有限公司 | Method for storing real-time data and system based on distributed parallel database |
| CN110134430A (en) * | 2019-04-12 | 2019-08-16 | 中国平安财产保险股份有限公司 | A kind of data packing method, device, storage medium and server |
| CN110347673A (en) * | 2019-05-30 | 2019-10-18 | 平安银行股份有限公司 | Data file loading method, device, computer equipment and storage medium |
| CN111144866A (en) * | 2019-12-25 | 2020-05-12 | 腾讯科技（深圳）有限公司 | Data transfer method, device, node equipment and storage medium |
| CN111324606A (en) * | 2020-01-23 | 2020-06-23 | 北京恒华伟业科技股份有限公司 | Data fragmentation method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11444866B2 (en) * | 2016-07-22 | 2022-09-13 | Intel Corporation | Methods and apparatus for composite node creation and management through SDI partitions |
-
2020
- 2020-08-27 CN CN202010879229.2A patent/CN112001160B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107967284A (en) * | 2016-10-20 | 2018-04-27 | 北京京东尚科信息技术有限公司 | Method and apparatus for storing, inquiring about sequence information |
| CN109189785A (en) * | 2018-08-10 | 2019-01-11 | 平安科技（深圳）有限公司 | Date storage method, device, computer equipment and storage medium |
| CN109669929A (en) * | 2018-12-14 | 2019-04-23 | 江苏瑞中数据股份有限公司 | Method for storing real-time data and system based on distributed parallel database |
| CN110134430A (en) * | 2019-04-12 | 2019-08-16 | 中国平安财产保险股份有限公司 | A kind of data packing method, device, storage medium and server |
| CN110347673A (en) * | 2019-05-30 | 2019-10-18 | 平安银行股份有限公司 | Data file loading method, device, computer equipment and storage medium |
| CN111144866A (en) * | 2019-12-25 | 2020-05-12 | 腾讯科技（深圳）有限公司 | Data transfer method, device, node equipment and storage medium |
| CN111324606A (en) * | 2020-01-23 | 2020-06-23 | 北京恒华伟业科技股份有限公司 | Data fragmentation method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112001160A (en) | 2020-11-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20110208947A1 (en) | System and Method for Simplifying Transmission in Parallel Computing System | |
| US11385830B2 (en) | Data storage method, apparatus and system, and server, control node and medium | |
| CN111580965A (en) | Data request processing method and system | |
| CN113420537B (en) | Method, device, equipment and storage medium for processing electronic form data | |
| CN112035144B (en) | Upgrading method and device of block chain system, computer equipment and storage medium | |
| CN111147392B (en) | Data synchronization method, device and equipment based on block chain and readable storage medium | |
| CN111506578A (en) | Service data checking method, device, equipment and storage medium | |
| CN112001160B (en) | Data processing method, device, equipment and storage medium | |
| CN112685612B (en) | Feature code searching and matching method, device and storage medium | |
| CN109462661B (en) | Data synchronization method and device, computer equipment and storage medium | |
| CN110297822B (en) | Block chain-oriented key management method, device, equipment and storage medium | |
| CN114896197A (en) | Method, device and system for uniformly storing multi-format documents | |
| CN111752761A (en) | Block chain big data safety processing method | |
| Konstantinidis et al. | Camr: Coded aggregated mapreduce | |
| CN111858467A (en) | File data processing method, device, equipment and medium based on artificial intelligence | |
| US11200204B2 (en) | Method, device and computer program product for searching a file | |
| CN111858764A (en) | Data synchronization method and device, computer equipment and storage medium | |
| Нікітін et al. | A collision-resistant hashing algorithm for maintaining consistency in distributed NoSQL databases | |
| CN116303286A (en) | Batch file processing method, device and system | |
| CN111787074B (en) | File synchronization method and terminal | |
| CN110489208B (en) | Virtual machine configuration parameter checking method, system, computer equipment and storage medium | |
| CN112559546A (en) | Database synchronization method and device, computer equipment and readable storage medium | |
| CN113612869A (en) | Short network address generation method, terminal and storage medium | |
| CN112015594A (en) | Backup method, device and equipment of rollback file and storage medium | |
| WO2023155644A1 (en) | Data processing method, forwarding chip, storage medium and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |