CN113127484A - Efficient and quick data storage method and device - Google Patents
Efficient and quick data storage method and device Download PDFInfo
- Publication number
- CN113127484A CN113127484A CN202011640446.2A CN202011640446A CN113127484A CN 113127484 A CN113127484 A CN 113127484A CN 202011640446 A CN202011640446 A CN 202011640446A CN 113127484 A CN113127484 A CN 113127484A
- Authority
- CN
- China
- Prior art keywords
- data
- cache server
- database
- storing
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000013500 data storage Methods 0.000 title claims abstract description 16
- 238000013507 mapping Methods 0.000 claims abstract description 21
- 238000004590 computer program Methods 0.000 claims description 7
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 238000013479 data entry Methods 0.000 claims 2
- 240000008100 Brassica rapa Species 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/31—Indexing; Data structures therefor; Storage structures
- G06F16/313—Selection or weighting of terms for indexing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of website data storage, in particular to a high-efficiency and quick data storage method and equipment. The method comprises the following steps: (1) receiving data to be stored input by a user; (2) storing the data into a cache server A on a server in a JSON text form; (3) analyzing the JSON text in the cache server A, and storing each field obtained by analysis into a middle table of a database; (4) when a field mapping command is received, determining a mapping relation according to the field mapping command; (5) and storing each field in the intermediate table into a product table of the database according to the mapping relation. According to the technical scheme, the waiting time of the user for submitting the data can be greatly reduced, and the efficiency of a data storage link is improved.
Description
Technical Field
The invention relates to the technical field of website data storage, in particular to a high-efficiency and quick data storage method and equipment.
Background
At present, a website usually needs to store mass text data into a database table, and the scheme in the prior art is to use a certain programming language to acquire data, extract a data conversion array by regular expression matching, and directly store the data into the database through the mapping relation between key values and fields of the array.
However, when the content to be inserted into the database reaches a certain level and the recorded content of the database reaches a certain level, the waiting time for the user to save the data will be very long, and meanwhile, if an interruption (such as error reporting) occurs, the data needs to be restored, the process is complex, and the experience for the user will be very poor.
Therefore, how to realize the fast mass storage of large data is a technical problem that needs to be solved by those skilled in the art at present.
Disclosure of Invention
Aiming at the defects of the technology, the invention aims to provide a high-efficiency and quick data storage method and equipment, which are used for realizing the batch and quick storage of large data.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
a high-efficiency and quick data storage method comprises the following steps:
(1) receiving data to be stored input by a user;
(2) respectively storing the data into a cache server A on a server in a JSON text form;
(3) analyzing the JSON text in the cache server A, and storing each field obtained by analysis into a middle table of a database;
(4) when a field mapping command is received, determining a mapping relation according to the field mapping command;
(5) and storing each field in the intermediate table into a product table of the database according to the mapping relation.
The JSON (JavaScript Object Notification) is a lightweight data exchange format, is based on a subset of ECMAScript (js specification established by European computer Association), adopts a text format completely independent of a programming language to store and represent data, and has a simple and clear hierarchy structure, so that the JSON becomes an ideal data exchange language, is easy to read and write by people, is easy to analyze and generate by a machine, and effectively improves the network transmission efficiency.
Further, the cache server a in the step (2) is used for storing data to be inserted into the database, and further includes a cache server B for reading data when being displayed at the front end.
Further, the data inserted in the cache server a is set to be unable to be cleared by the user, and the data of the cache server B is set to be cleared by the user.
Further, after the data in the cache server B is cleared, the step of rewriting the data into the cache is as follows:
1) reading data needing to be rewritten from a database;
2) reading data corresponding to the step 1) from a cache server A;
3) merging and sequencing the data read in the database and the data read in the cache server A to obtain new data;
4) storing the new data obtained after merging and sorting in the step 3) into the current cache server B again.
Further, when the data in the cache server a reaches a set critical value, a Socket tool is used for pushing a message which can be put in storage and delete the corresponding cache data to the cache server a.
Further, the use time of the data storage method can be set, and the specific method comprises the following steps:
1) the method comprises the steps that a user acquires a self-increment primary key value of current data in a database when storing data records into the database every time;
2) recording the obtained self-increment primary key values into a cache server A;
3) the user acquires the corresponding self-increment primary key value recorded in the cache server A every time the user stores data, and the method is started when the self-increment primary key value reaches a set critical value.
The application also includes a data warehousing device, the data warehousing device including:
a memory for storing a computer program;
and the processor is used for realizing the steps of any one of the data warehousing methods when the computer program is executed.
The technical effects of the invention are as follows: the waiting time of the user for submitting the data is greatly reduced, and the efficiency of the data storage link is greatly improved; meanwhile, the server and the database are used as media, so that the cooperative work of different development tools in the same environment is realized, and other links of the original system are not influenced. The application also provides equipment for data storage, and the equipment has the effects of improving user experience and improving system performance.
Drawings
FIG. 1 is a flow chart of a data warehousing method in this embodiment;
Detailed Description
The following is further detailed by way of specific embodiments:
the core of the invention is to provide a high-efficiency and quick data storage method, a system, equipment and a computer readable storage medium, which are used for realizing the batch and quick storage of large data.
As shown in fig. 1, a high-efficiency and fast data storage method includes the following steps:
(1) receiving data to be stored input by a user;
(2) converting data to be stored into a JSON text by using a programming tool, and storing the JSON text to a cache server, wherein the cache server comprises a cache server A and a cache server B, the cache server A is used for storing the data to be inserted into a database, the cache server B reads the data when being displayed by a front end, the data inserted into the cache server A is set to be incapable of being manually cleared by a user, and the data of the cache server B is set to be manually cleared by the user;
and (3) converting the data to be stored into the JSON text in the step (2) by using a PHP (graphical user protocol) programming tool, wherein the PHP is a universal open source scripting language, is high in execution efficiency, simple in deployment, low in development cost and widely applied to domestic website development. The method for converting the data to be stored into the JSON text comprises the following steps: inputting a regular expression; extracting data to be stored through a regular expression by using a PHP tool to obtain required data; and converting the required data into JSON text by utilizing a PHP tool.
(3) Analyzing the JSON text in the cache server A by using a programming tool (such as Go, Rube and Python), and storing each field obtained by analysis into an intermediate table of a database;
(4) when a field mapping command is received, determining a mapping relation according to the field mapping command;
(5) storing each field in the intermediate table of the database into a product table of the database according to a mapping relation by using a programming tool (such as Go, Rube and Python) and clearing corresponding cache content in the cache server A; in the preferred mode in this step, a Socket or other tools are used for pushing each time data is inserted, and when the data amount in the cache server a reaches a critical value, the server is notified to perform a storage operation and delete the corresponding cache data, so as to reduce the pressure of the cache server a.
Furthermore, in order to improve the utilization efficiency of the data storage method, the use timing can be set by the following method:
1) the method comprises the steps that a user acquires a self-increment primary key value of current data in a database when storing data records into the database every time;
2) recording the obtained self-increment primary key values into a cache server A;
3) the user acquires the corresponding self-increment primary key value recorded in the cache server A every time the user stores data, and the method is started when the self-increment primary key value reaches a set critical value.
The cache server B reads data when being displayed by the front end, if the cache data in the cache server B is empty, the cache server B reads corresponding data in the database firstly, then reads corresponding cache data in the cache server A, then combines and sorts the two data to generate final data, and finally caches the final data in the cache server B to be read when being displayed by the front end.
Further, after the data in the cache server B is cleared, the step of rewriting the data into the cache is as follows:
1) reading data needing to be rewritten from a database;
2) reading data corresponding to the step 1) from a cache server A;
3) merging and sequencing the data read in the database and the data read in the cache server A to obtain new data;
4) storing the new data obtained after merging and sorting in the step 3) into the current cache server B again.
Based on the technical scheme, the method for warehousing the data, provided by the application, comprises the steps of extracting required information in the data to be stored through a regular expression, converting the required information into a JSON text and storing the JSON text into a cache server A, analyzing the JSON text on the cache server A through Go, Rube or Python tools, storing each field obtained through analysis into an intermediate table of a database, determining a mapping relation according to a field mapping command when the field mapping command is received, storing each field in the intermediate table of the database into a product table of the database according to the mapping relation through Go, Rube, Python and other programming tools, and clearing corresponding cache contents in the cache server A, so that the waiting time of a user for submitting the data is greatly reduced, and the efficiency of a data warehousing link is greatly improved; meanwhile, the server and the database are used as media, so that the cooperative work of different development tools in the same environment is realized, and other links of the original system are not influenced.
The embodiment also provides a data warehousing device, which comprises a memory and a processor, wherein the memory is used for storing the computer program; the processor is used for realizing the steps of the data warehousing method when executing the computer program. It is clear to those skilled in the art that, for convenience and brevity of description, the specific working process of the apparatus described above may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.
The technical solution of the present application, or a part or all or part of the technical solution that contributes to the prior art, may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which may be a personal computer, a function calling device, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk. Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer program code for carrying out operations of the present invention may be written in one or more programming languages, or a combination thereof, such as C + +, C, JAVA, or the like, in addition to the programming tools described above.
The foregoing is considered as illustrative of the preferred embodiments of the invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims. In addition, for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts, and the application can be subjected to a plurality of improvements and modifications without departing from the principle of the application, and the improvements and modifications also fall into the protection scope of the claims of the application.
Claims (7)
1. A high-efficiency and quick data storage method is characterized by comprising the following steps:
(1) receiving data to be stored input by a user;
(2) storing the data into a cache server A on a server in a JSON text form;
(3) analyzing the JSON text in the cache server A, and storing each field obtained by analysis into a middle table of a database;
(4) when a field mapping command is received, determining a mapping relation according to the field mapping command;
(5) and storing each field in the intermediate table into a product table of the database according to the mapping relation.
2. An efficient and fast data warehousing method as claimed in claim 1, wherein in the step (2), the cache server a is used to store data to be inserted into the database, and further comprises a cache server B, and the cache server B uses data read when being displayed at the front end.
3. An efficient and fast data warehousing method as claimed in claim 2, characterized in that the data inserted into the cache server a is set to be unable to be manually cleared by the user, and the data of the cache server B is set to be manually cleared by the user.
4. A method as claimed in claim 3, wherein the step of rewriting the data into the cache after the data in the cache server B is removed is as follows:
1) reading data needing to be rewritten from a database;
2) reading data corresponding to the step 1) from a cache server A;
3) merging and sequencing the data read in the database and the data read in the cache server A to obtain new data;
4) storing the new data obtained after merging and sorting in the step 3) into the current cache server B again.
5. The efficient and fast data warehousing method as claimed in claim 1, wherein when the data in the cache server a reaches a set critical value, a Socket tool is used to push a message to the cache server a for warehousing and deleting the corresponding cache data.
6. An efficient and fast data entry method as claimed in claim 1, wherein the time for using the data entry method can be set, and the specific method is as follows:
1) the method comprises the steps that a user acquires a self-increment primary key value of current data in a database when storing data records into the database every time;
2) recording the obtained self-increment primary key values into a cache server A;
3) the user acquires the corresponding self-increment primary key value recorded in the cache server A every time the user stores data, and the method is started when the self-increment primary key value reaches a set critical value.
7. A data warehousing device characterized by: comprising a memory and a processor, wherein the memory is used for storing a computer program; the processor is adapted to carry out the steps of the method of data warehousing as claimed in any one of claims 1 to 6 when the computer program is executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011640446.2A CN113127484A (en) | 2020-12-31 | 2020-12-31 | Efficient and quick data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011640446.2A CN113127484A (en) | 2020-12-31 | 2020-12-31 | Efficient and quick data storage method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113127484A true CN113127484A (en) | 2021-07-16 |
Family
ID=76772359
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011640446.2A Pending CN113127484A (en) | 2020-12-31 | 2020-12-31 | Efficient and quick data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113127484A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104572860A (en) * | 2014-12-17 | 2015-04-29 | 北京皮尔布莱尼软件有限公司 | Data processing method and data processing system |
| CN106506663A (en) * | 2016-11-22 | 2017-03-15 | 北京科摩仕捷科技有限公司 | A kind of number distributing method, server and system |
| CN107038207A (en) * | 2017-02-20 | 2017-08-11 | 阿里巴巴集团控股有限公司 | A kind of data query method, data processing method and device |
| CN107633014A (en) * | 2017-08-17 | 2018-01-26 | 深圳市诚壹科技有限公司 | A kind of date storage method and server |
| CN109151061A (en) * | 2018-09-28 | 2019-01-04 | 视联动力信息技术股份有限公司 | A kind of date storage method and device |
| CN110365752A (en) * | 2019-06-27 | 2019-10-22 | 北京大米科技有限公司 | Processing method, device, electronic equipment and the storage medium of business datum |
| CN110502520A (en) * | 2019-08-27 | 2019-11-26 | 云汉芯城（上海）互联网科技股份有限公司 | A kind of method of data loading, system, equipment and computer readable storage medium |
-
2020
- 2020-12-31 CN CN202011640446.2A patent/CN113127484A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104572860A (en) * | 2014-12-17 | 2015-04-29 | 北京皮尔布莱尼软件有限公司 | Data processing method and data processing system |
| CN106506663A (en) * | 2016-11-22 | 2017-03-15 | 北京科摩仕捷科技有限公司 | A kind of number distributing method, server and system |
| CN107038207A (en) * | 2017-02-20 | 2017-08-11 | 阿里巴巴集团控股有限公司 | A kind of data query method, data processing method and device |
| CN107633014A (en) * | 2017-08-17 | 2018-01-26 | 深圳市诚壹科技有限公司 | A kind of date storage method and server |
| CN109151061A (en) * | 2018-09-28 | 2019-01-04 | 视联动力信息技术股份有限公司 | A kind of date storage method and device |
| CN110365752A (en) * | 2019-06-27 | 2019-10-22 | 北京大米科技有限公司 | Processing method, device, electronic equipment and the storage medium of business datum |
| CN110502520A (en) * | 2019-08-27 | 2019-11-26 | 云汉芯城（上海）互联网科技股份有限公司 | A kind of method of data loading, system, equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110502520B (en) | Method, system and equipment for data warehousing and computer readable storage medium | |
| CN111367524B (en) | Enumeration type design method and device | |
| CN109872230B (en) | Test method and device of financial data analysis system, medium and electronic equipment | |
| CN115878003B (en) | Method and system for automating RPA webpage operation based on Transformer | |
| CN112231407B (en) | DDL synchronization method, device, equipment and medium of PostgreSQL database | |
| CN113076167A (en) | Code processing method and related equipment | |
| CN114138244A (en) | Method and device for automatically generating model files, storage medium and electronic equipment | |
| CN114036183A (en) | Data ETL processing method, device, equipment and medium | |
| CN116227505A (en) | Internationalization file translation method, device, equipment and medium | |
| CN103226510A (en) | Method and device for analyzing vmcore file | |
| CN112231197A (en) | Page testing method and device and storage medium | |
| CN111783401B (en) | Character string processing method and device, computer storage medium and electronic equipment | |
| CN113760894A (en) | Data calling method and device, electronic equipment and storage medium | |
| CN111026604B (en) | Log file analysis method and device | |
| CN109684207B (en) | Method and device for packaging operation sequence, electronic equipment and storage medium | |
| CN113127484A (en) | Efficient and quick data storage method and device | |
| CN115686493A (en) | Method for automatically generating interface control document by C language data structure code | |
| CN112861138A (en) | Software security analysis method and analysis device, electronic device, and storage medium | |
| CN116150209A (en) | Report computing system, method, electronic equipment and storage medium | |
| CN112765110B (en) | PDF annotation data generation method, device, equipment and storage medium | |
| CN109344392B (en) | Intelligent message pushing method, system and device for security customer service consultation | |
| CN114416107A (en) | Method, device, storage medium and equipment for translating logic | |
| CN111241191A (en) | Database synchronization method and device | |
| CN110515913B (en) | Log processing method and device | |
| CN110147364B (en) | Data cleaning method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210716 |