CN110825731A - Data storage method and device, electronic equipment and storage medium - Google Patents
Data storage method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN110825731A CN110825731A CN201910881337.0A CN201910881337A CN110825731A CN 110825731 A CN110825731 A CN 110825731A CN 201910881337 A CN201910881337 A CN 201910881337A CN 110825731 A CN110825731 A CN 110825731A
- Authority
- CN
- China
- Prior art keywords
- data
- buried point
- point data
- target
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000013500 data storage Methods 0.000 title claims abstract description 45
- 238000005516 engineering process Methods 0.000 claims description 11
- 238000009933 burial Methods 0.000 claims 2
- 238000012545 processing Methods 0.000 abstract description 10
- 238000012216 screening Methods 0.000 abstract description 6
- 238000004590 computer program Methods 0.000 description 14
- 230000008569 process Effects 0.000 description 14
- 238000007405 data analysis Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 8
- 230000006399 behavior Effects 0.000 description 6
- 238000011161 development Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 4
- 238000013178 mathematical model Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000006386 memory function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention provides a data storage method, a data storage device, electronic equipment and a storage medium. The data storage method can write the data of the buried points into a message queue when the data of the buried points are received, realize the temporary storage of the data, avoid influencing the normal operation speed of a system, and obtain the data of the target buried points of preset types from the message queue, thereby realizing the secondary screening of the data, avoiding the redundancy of the data, further analyzing the data of the target buried points, generating a data table, and storing the data table in a configuration database, thereby realizing the asynchronous storage of the relational data, avoiding causing the operation pressure of the system, processing the data, effectively improving the operation performance of the system, and being convenient for the use of the subsequent data because the relation among all the data in the stored data is more definite.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data storage method and apparatus, an electronic device, and a storage medium.
Background
In the prior art, the data storage granularity of most products in the industry is coarse, data are simply mixed together, when a user wants to search required data from the stored data, the user needs to search all the data, the required data are difficult to search due to disordered relation among the data, and the output of a data report and data analysis are not facilitated.
If the data is directly stored in the server in use while the data is acquired, although the problem of coarse granularity of data storage can be solved, the server runs slowly due to the fact that the memory occupied in the storage process is high, and normal execution of other data is affected.
Therefore, the problem of coarse data storage granularity cannot be solved under the condition that normal operation of the server is not influenced at present.
Disclosure of Invention
In view of the above, it is necessary to provide a data storage method, an apparatus, an electronic device, and a storage medium, which can implement asynchronous storage of relational data, avoid causing system operation pressure, and effectively improve system operation performance.
A method of data storage, the method comprising:
when data of buried points is received, writing the data of the buried points into a message queue;
acquiring preset type target buried point data from the message queue;
analyzing the target buried point data to generate a data table;
storing the data table in a configuration database.
According to a preferred embodiment of the invention, the method further comprises:
when a login signal is detected, acquiring all data operation types;
displaying all the data operation types;
when a selection signal of any operation type in all the data operation types is received, acquiring first data corresponding to the selected first operation type;
and determining the first data as the buried point data.
According to a preferred embodiment of the invention, when the buried point data is received, the method further comprises:
calling a configuration interface;
connecting to a configuration server through the configuration interface;
and sending the buried point data to the configuration server.
According to a preferred embodiment of the present invention, the obtaining of the preset type of target buried point data from the message queue includes one or more of the following combinations:
displaying all sub-operation types of the first operation type, when a selection signal of any operation type in all the sub-operation types is received, acquiring second data corresponding to the selected preset type, and determining the second data as the target buried point data; and/or
And determining the operation type of the mark as the preset type, acquiring second data corresponding to the preset type, and determining the second data as the target buried point data.
According to a preferred embodiment of the present invention, the analyzing the target buried point data and generating a data table includes:
and connecting at least one server, and analyzing the target buried point data by the at least one server to generate a data table.
According to a preferred embodiment of the present invention, the analyzing the target buried point data and generating a data table further includes:
analyzing the target buried point data by adopting a FastJson technology or a GSON technology to obtain the relation among all data in the target buried point data;
and establishing the data table based on the relation among all data in the target buried point data.
According to a preferred embodiment of the present invention, when storing the data table in a configuration database, the method further comprises:
detecting other data to be stored except the data table;
and when detecting that other data to be stored exist, storing the other data to be stored and the data table in the configuration database at the same time.
A data storage device, the device comprising:
the writing unit is used for writing the buried point data into a message queue when the buried point data is received;
the acquisition unit is used for acquiring preset type target buried point data from the message queue;
the generating unit is used for analyzing the target buried point data to generate a data table;
and the storage unit is used for storing the data table in a configuration database.
According to the preferred embodiment of the present invention, the obtaining unit is further configured to obtain all data operation types when a login signal is detected;
the device further comprises:
the display unit is used for displaying all the data operation types;
the acquiring unit is further configured to acquire first data corresponding to a selected first operation type when a selection signal of any operation type in all the data operation types is received;
a determination unit configured to determine the first data as the buried point data.
According to a preferred embodiment of the invention, the apparatus further comprises:
the calling unit is used for calling the configuration interface when receiving the buried point data;
the calling unit is also used for being connected to a configuration server through the configuration interface;
and the sending unit is used for sending the buried point data to the configuration server.
According to a preferred embodiment of the present invention, the obtaining unit obtains the preset type of target buried point data from the message queue by one or a combination of the following manners:
displaying all sub-operation types of the first operation type, when a selection signal of any operation type in all the sub-operation types is received, acquiring second data corresponding to the selected preset type, and determining the second data as the target buried point data; and/or
And determining the operation type of the mark as the preset type, acquiring second data corresponding to the preset type, and determining the second data as the target buried point data.
According to a preferred embodiment of the present invention, the generating unit is specifically configured to:
and connecting at least one server, and analyzing the target buried point data by the at least one server to generate a data table.
According to a preferred embodiment of the present invention, the generating unit is further specifically configured to:
analyzing the target buried point data by adopting a FastJson technology or a GSON technology to obtain the relation among all data in the target buried point data;
and establishing the data table based on the relation among all data in the target buried point data.
According to a preferred embodiment of the invention, the apparatus further comprises:
the detection unit is used for detecting other data to be stored except the data table when the data table is stored in a configuration database;
the storage unit is further configured to store the other data to be stored and the data table in the configuration database simultaneously when detecting that there is other data to be stored.
An electronic device, the electronic device comprising:
a memory storing at least one instruction; and
a processor executing instructions stored in the memory to implement the data storage method.
A computer-readable storage medium having stored therein at least one instruction, the at least one instruction being executable by a processor in an electronic device to implement the data storage method.
According to the technical scheme, when the data of the buried point is received, the data of the buried point is written into the message queue, the temporary storage of the data is realized, the influence on the normal operation speed of the system is avoided, the preset type of target data of the buried point is obtained from the message queue, the secondary screening of the data is realized, the redundancy of the data is avoided, the target data of the buried point is further analyzed, the data table is generated, and the data table is stored in the configuration database, so that the asynchronous storage of the relational data is realized, the system operation pressure is avoided, the operation performance of the system is effectively improved, and the use of the subsequent data is facilitated because the relation among all data in the stored data is more definite.
Drawings
FIG. 1 is a flow chart of a preferred embodiment of the data storage method of the present invention.
FIG. 2 is a functional block diagram of a preferred embodiment of a data storage device according to the present invention.
FIG. 3 is a schematic structural diagram of an electronic device implementing a data storage method according to a preferred embodiment of the invention.
Description of the main elements
| |
1 |
| |
12 |
| Processor with a memory having a plurality of |
13 |
| Data storage device | 11 |
| |
110 |
| |
111 |
| |
112 |
| |
113 |
| |
114 |
| Determining |
115 |
| |
116 |
| |
117 |
| |
118 |
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in detail with reference to the accompanying drawings and specific embodiments.
FIG. 1 is a flow chart of a data storage method according to a preferred embodiment of the present invention. The order of the steps in the flow chart may be changed and some steps may be omitted according to different needs.
The data storage method is applied to one or more electronic devices, which are devices capable of automatically performing numerical calculation and/or information processing according to preset or stored instructions, and the hardware thereof includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The electronic device may be any electronic product capable of performing human-computer interaction with a user, for example, a Personal computer, a tablet computer, a smart phone, a Personal Digital Assistant (PDA), a game machine, an interactive Internet Protocol Television (IPTV), an intelligent wearable device, and the like.
The electronic device may also include a network device and/or a user device. The network device includes, but is not limited to, a single network server, a server group consisting of a plurality of network servers, or a cloud computing (cloud computing) based cloud consisting of a large number of hosts or network servers.
The Network where the electronic device is located includes, but is not limited to, the internet, a wide area Network, a metropolitan area Network, a local area Network, a Virtual Private Network (VPN), and the like.
And S10, when the buried point data is received, writing the buried point data into a message queue.
In at least one embodiment of the present invention, a buried point refers to a related technology and its implementation process for capturing, processing and transmitting specific user behavior or events. The embedding points are used for embedding points for each event of user behaviors for the requirement of statistical analysis, and the data results are analyzed to further optimize products or guide operation.
In at least one embodiment of the present invention, the buried point data refers to data generated in the above process.
Thus, the buried point data is generated for a specified operation behavior, i.e., the buried point data corresponds to at least one agreed data operation type.
In at least one embodiment of the invention, the method further comprises:
when a login signal is detected, the electronic equipment acquires all data operation types, further, the electronic equipment displays all the data operation types, when a selection signal of any operation type in all the data operation types is received, the electronic equipment acquires first data corresponding to the selected first operation type, and further, the electronic equipment determines the first data as the buried point data.
For example: (1) when the operation of position forwarding on the configuration interface is detected, the electronic equipment determines the data generated by the position forwarding operation as the buried point data because the position forwarding operation belongs to the selected first operation type.
(2) When the operation of checking the working position is detected on the configuration interface, the electronic equipment determines the data generated by the operation of checking the working position as the buried point data because the operation of checking the working position belongs to the selected first operation type.
Wherein the configuration interface may include, but is not limited to: job hunting interfaces, push-in interfaces, etc., the present invention is not limited.
Through the embodiment, the buried point data can be determined according to actual service requirements, and data redundancy caused by redundant data storage is avoided.
In at least one embodiment of the present invention, the Message Queue (MQ) is an important component in a distributed system, mainly solves the problems of application decoupling, asynchronous messages, traffic cut-off, and the like, and implements a high-performance, high-availability, scalable, and final consistency architecture. Currently, more message queues are used including, but not limited to: ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocktMQ, and the like.
Through the message queue, programs can communicate by sending data in messages instead of directly calling each other, so that the data embedded can be temporarily stored, and the influence on the execution of other data caused by the storage of the electronic equipment can be avoided, namely, the problem of the running performance of the electronic equipment can be effectively solved through a processing mode of asynchronous storage.
In at least one embodiment of the invention, when the electronic device receives the buried point data, the method further comprises:
and the electronic equipment calls a configuration interface, is connected to a configuration server through the configuration interface, and sends the buried point data to the configuration server.
Specifically, the configuration interface is used to invoke the configuration server, and the design manner of the configuration interface is not limited in the present invention.
Specifically, the configuration server is a server for performing front-end point burying.
Further, when the electronic device detects that there is an operation of the first operation type on one page, the electronic device obtains first data corresponding to the first operation type, and the electronic device directly sends the first data to the configuration server, without participating in a data transmission process, and the first data is original data and is not subjected to any data processing, so that storage of the obtained original data is realized, and the obtained original data belongs to a coarse-grained data storage process.
The data type of the buried point data is in a JSON (JavaScript Object Notation) data exchange format.
Further, the JSON data exchange format is a lightweight data exchange language, and stores and represents data in a text format that is completely independent of a programming language.
According to the embodiment, the electronic equipment is not required to be used for storing data, the memory of the electronic equipment is saved, the storage process can be used for storing all data, and the stored data can be used as backup and can be used for executing other operations although the stored data are not convenient to analyze.
Moreover, through the implementation mode, the stored data can be more comprehensive, the data loss can be effectively avoided, and meanwhile, the running performance of the system cannot be influenced.
And S11, acquiring preset type target buried point data from the message queue.
In at least one embodiment of the present invention, the target buried point data refers to data that is determined according to actual requirements and needs to be subjected to data relationship analysis before being stored.
That is, the target buried point data is the preset type of data stored in the message queue.
The preset type may be agreed and marked before data storage according to actual service requirements, or determined by a user, which is not limited in the present invention.
For example: and marking the development position forwarding operation as the preset type of target buried point data according to the prearranged position forwarding operation type of the buried point data.
In at least one embodiment of the present invention, the target buried point data is used as basic data for data analysis, and further screening determination is required to be performed on the buried point data.
In at least one embodiment of the present invention, the electronic device obtains the preset type of target buried point data from the message queue by one or more of the following methods:
(1) and the electronic equipment displays all the sub-operation types of the first operation type, acquires second data corresponding to the selected preset type when a selection signal of any operation type in all the sub-operation types is received, and further determines the second data as the target buried point data.
(2) And the electronic equipment determines the operation type of the mark as the preset type, acquires second data corresponding to the preset type, and further determines the second data as the target buried point data.
For example: the electronic equipment further acquires a development position forwarding operation from all sub-operation types of the first operation type as the preset type, acquires second data generated by the development position forwarding operation, and further determines the second data as the target buried point data.
Through the above embodiment, the electronic device can further screen the first data corresponding to the first operation type, and then obtain the second data as the target buried point data.
It can be understood that, since some data have no substantial meaning to the data analysis, query, and other types of operations, the redundant data can be skipped directly, and the electronic device does not need to analyze all data in the buried point data, thereby avoiding redundancy of data, causing a burden on storage of a database, and being not beneficial to call of subsequent data.
And S12, analyzing the target buried point data to generate a data table.
In at least one embodiment of the invention, the data table is a binary data table in which column data is fixed and row data is different.
By analyzing the target buried point data, the relational data, namely the data table, can be obtained, and because the relevance among the data is high and the relation among the data is clear, the operations such as data query and data analysis can be conveniently executed, the results of the data query and the data analysis are more accurate, and the usability of the data is improved.
In at least one embodiment of the present invention, the analyzing, by the electronic device, the target buried point data, and generating the data table includes:
the electronic equipment is connected with at least one server, and the target buried point data is analyzed by the at least one server to generate a data table.
Specifically, the at least one server refers to a server in communication with the electronic device.
By the implementation mode, the target buried point data can be analyzed in parallel by the plurality of servers, the memory of the electronic equipment is not occupied, the data execution efficiency is improved, and the execution of other data in the electronic equipment is effectively prevented from being influenced.
Of course, in other embodiments, in view of resource limitations, the electronic device may analyze the target buried point data by itself if there is not enough servers as a support, and the present invention is not limited as long as the data analysis before storage can be achieved.
Specifically, the analyzing, by the electronic device, the target buried point data, and generating the data table further includes:
the electronic equipment analyzes the target buried point data by adopting a FastJson technology or a GSON technology to obtain the relation among all data in the target buried point data, and further, the electronic equipment establishes the data table based on the relation among all data in the target buried point data.
In the above embodiment, the data table can be generated, the data table is a relational data, the relational data is represented by a relational mathematical model, and the data is described in the form of a two-dimensional table in the relational mathematical model, so that the data can be stored, and the relationship between each piece of data in the stored data can be clarified, thereby facilitating the use of subsequent data.
S13, storing the data table in a configuration database.
In at least one embodiment of the present invention, the data table stores fine-grained data in the configuration database, and belongs to a data storage process of a back-end buried point. In addition, due to the fact that asynchronous storage is carried out, the electronic equipment is not executed when data are received, and therefore execution of other data in the process cannot be influenced, system operation pressure is avoided, and operation performance of the system is effectively improved.
In at least one embodiment of the invention, the configuration database may be any database, and the invention is not limited.
Of course, to avoid affecting the operating speed of the electronic device, the configuration database may preferably be an external database in communication with the electronic device, such that the operation of the electronic device is not affected regardless of the amount of storage in the configuration database.
In at least one embodiment of the invention, when storing the data table in a configuration database, the method further comprises:
and the electronic equipment detects other data to be stored except the data table, and stores the other data to be stored and the data table in the configuration database simultaneously when detecting that other data to be stored exist.
Through the embodiment, the asynchronous storage of the data table can be realized, and the batch storage of the data can be realized, so that the data storage efficiency is improved, and the data blockage is effectively avoided.
According to the technical scheme, when the data of the buried point is received, the data of the buried point is written into the message queue, the temporary storage of the data is realized, the influence on the normal operation speed of the system is avoided, the preset type of target data of the buried point is obtained from the message queue, the secondary screening of the data is realized, the redundancy of the data is avoided, the target data of the buried point is further analyzed, the data table is generated, and the data table is stored in the configuration database, so that the asynchronous storage of the relational data is realized, the system operation pressure is avoided, the operation performance of the system is effectively improved, and the use of the subsequent data is facilitated because the relation among all data in the stored data is more definite.
FIG. 2 is a functional block diagram of a data storage device according to a preferred embodiment of the present invention. The data storage device 11 includes a writing unit 110, an acquisition unit 111, a generation unit 112, a storage unit 113, a display unit 114, a determination unit 115, a calling unit 116, a sending unit 117, and a detection unit 118. The module/unit referred to in the present invention refers to a series of computer program segments that can be executed by the processor 13 and that can perform a fixed function, and that are stored in the memory 12. In the present embodiment, the functions of the modules/units will be described in detail in the following embodiments.
When the buried point data is received, the writing unit 110 writes the buried point data into the message queue.
In at least one embodiment of the present invention, a buried point refers to a related technology and its implementation process for capturing, processing and transmitting specific user behavior or events. The embedding points are used for embedding points for each event of user behaviors for the requirement of statistical analysis, and the data results are analyzed to further optimize products or guide operation.
In at least one embodiment of the present invention, the buried point data refers to data generated in the above process.
Thus, the buried point data is generated for a specified operation behavior, i.e., the buried point data corresponds to at least one agreed data operation type.
In at least one embodiment of the invention, the method further comprises:
when a login signal is detected, the obtaining unit 111 obtains all data operation types, further, the display unit 114 displays all data operation types, when a selection signal for any operation type in all data operation types is received, the obtaining unit 111 obtains first data corresponding to the selected first operation type, and further, the determining unit 115 determines the first data as the buried point data.
For example: (1) when an operation of job forwarding on the configuration interface is detected, since the job forwarding operation belongs to the selected first operation type, the determining unit 115 determines data generated by the job forwarding operation as the buried point data.
(2) When an operation of viewing a work position is detected on the configuration interface, since the operation of viewing a work position belongs to the selected first operation type, the determining unit 115 determines data generated by the operation of viewing a work position as the buried point data.
Wherein the configuration interface may include, but is not limited to: job hunting interfaces, push-in interfaces, etc., the present invention is not limited.
Through the embodiment, the buried point data can be determined according to actual service requirements, and data redundancy caused by redundant data storage is avoided.
In at least one embodiment of the present invention, the Message Queue (MQ) is an important component in a distributed system, mainly solves the problems of application decoupling, asynchronous messages, traffic cut-off, and the like, and implements a high-performance, high-availability, scalable, and final consistency architecture. Currently, more message queues are used including, but not limited to: ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocktMQ, and the like.
Through the message queue, the programs can communicate by sending data in the message instead of directly calling each other, so that the data of the embedded point can be temporarily stored, and the influence on the execution of other data caused by the storage of the electronic equipment can be avoided, namely, the problem of the running performance of the electronic equipment can be effectively solved through the processing mode of asynchronous storage.
In at least one embodiment of the present invention, when the writing unit 110 receives the buried point data, the method further includes:
the calling unit 116 calls a configuration interface through which to connect to a configuration server, and further, the sending unit 117 sends the buried point data to the configuration server.
Specifically, the configuration interface is used to invoke the configuration server, and the design manner of the configuration interface is not limited in the present invention.
Specifically, the configuration server is a server for performing front-end point burying.
Further, that is, when the detecting unit 118 detects that there is an operation of the first operation type on one page, the obtaining unit 111 obtains first data corresponding to the first operation type, and the sending unit 117 directly sends the first data to the configuration server, without the electronic device participating in a data transmission process, and the first data is original data and is not subjected to any data processing, so that what is achieved is that the obtained original data is stored, and belongs to a coarse-grained data storage process.
The data type of the buried point data is in a JSON (JavaScript Object Notation) data exchange format.
Further, the JSON data exchange format is a lightweight data exchange language, and stores and represents data in a text format that is completely independent of a programming language.
According to the embodiment, the electronic equipment is not required to be used for storing data, the memory of the electronic equipment is saved, the storage process can be used for storing all data, and the stored data can be used as backup and can be used for executing other operations although the stored data are not convenient to analyze.
Moreover, through the implementation mode, the stored data can be more comprehensive, the data loss can be effectively avoided, and meanwhile, the running performance of the system cannot be influenced.
The obtaining unit 111 obtains the preset type of target buried point data from the message queue.
In at least one embodiment of the present invention, the target buried point data refers to data that is determined according to actual requirements and needs to be subjected to data relationship analysis before being stored.
That is, the target buried point data is the preset type of data stored in the message queue.
The preset type may be agreed and marked before data storage according to actual service requirements, or determined by a user, which is not limited in the present invention.
For example: and marking the development position forwarding operation as the preset type of target buried point data according to the prearranged position forwarding operation type of the buried point data.
In at least one embodiment of the present invention, the target buried point data is used as basic data for data analysis, and further screening determination is required to be performed on the buried point data.
In at least one embodiment of the present invention, the obtaining unit 111 obtains the preset type of target buried point data from the message queue by one or more of the following ways:
(1) the obtaining unit 111 displays all sub-operation types of the first operation type, and when a selection signal for any operation type of the all sub-operation types is received, the obtaining unit 111 further obtains second data corresponding to the selected preset type, and determines the second data as the target buried point data.
(2) The obtaining unit 111 determines an operation type of the mark as the preset type, obtains second data corresponding to the preset type, and further, the obtaining unit 111 determines the second data as the target buried point data.
For example: the obtaining unit 111 further obtains a development position forwarding operation from all sub-operation types of the first operation type as the preset type, and the obtaining unit 111 obtains second data generated by the development position forwarding operation, and determines the second data as the target buried point data.
With the above embodiment, the obtaining unit 111 can further screen the first data corresponding to the first operation type, and then obtain the second data as the target buried point data.
It can be understood that, since some data have no substantial meaning to the data analysis and query operations, the redundant data can be skipped directly, and the generating unit 112 does not need to analyze all data in the buried point data, thereby avoiding redundancy of data, causing a burden on storage of the database, and being not beneficial to call of subsequent data.
The generation unit 112 analyzes the target buried point data to generate a data table.
In at least one embodiment of the invention, the data table is a binary data table in which column data is fixed and row data is different.
By analyzing the target buried point data, the relational data, namely the data table, can be obtained, and because the relevance among the data is high and the relation among the data is clear, the operations such as data query and data analysis can be conveniently executed, the results of the data query and the data analysis are more accurate, and the usability of the data is improved.
In at least one embodiment of the present invention, the generating unit 112 analyzes the target buried point data, and generating a data table includes:
the generating unit 112 is connected to at least one server, and analyzes the target buried point data with the at least one server to generate a data table.
Specifically, the at least one server refers to a server in communication with the electronic device.
By the implementation mode, the target buried point data can be analyzed in parallel by the plurality of servers, the memory of the electronic equipment is not occupied, the data execution efficiency is improved, and the execution of other data in the electronic equipment is effectively prevented from being influenced.
Of course, in other embodiments, in view of resource limitations, if there is not enough servers as a support, the generating unit 112 may analyze the target buried point data by itself, and the present invention is not limited as long as the data analysis before storage can be implemented.
Specifically, the generating unit 112 analyzes the target buried point data, and generating a data table further includes:
the generating unit 112 analyzes the target buried point data by using a FastJson technique or a GSON technique to obtain a relationship between each data in the target buried point data, and further, the generating unit 112 establishes the data table based on the relationship between each data in the target buried point data.
In the above embodiment, the data table can be generated, the data table is a relational data, the relational data is represented by a relational mathematical model, and the data is described in the form of a two-dimensional table in the relational mathematical model, so that the data can be stored, and the relationship between each piece of data in the stored data can be clarified, thereby facilitating the use of subsequent data.
The storage unit 113 stores the data table in a configuration database.
In at least one embodiment of the present invention, the data table stores fine-grained data in the configuration database, and belongs to a data storage process of a back-end buried point. In addition, because asynchronous storage is performed and the data is not executed when the data is received, the execution of other data in the process cannot be influenced, the running pressure of the system is avoided, and the running performance of the system is effectively improved.
In at least one embodiment of the invention, the configuration database may be any database, and the invention is not limited.
Of course, to avoid affecting the operating speed of the electronic device, the configuration database may preferably be an external database in communication with the electronic device, such that the operation of the electronic device is not affected regardless of the amount of storage in the configuration database.
In at least one embodiment of the invention, when storing the data table in a configuration database, the method further comprises:
the detecting unit 118 detects other data to be stored except the data table, and when other data to be stored is detected, the storing unit 113 stores the other data to be stored and the data table in the configuration database at the same time.
Through the embodiment, the asynchronous storage of the data table can be realized, and the batch storage of the data can be realized, so that the data storage efficiency is improved, and the data blockage is effectively avoided.
According to the technical scheme, when the data of the buried point is received, the data of the buried point is written into the message queue, the temporary storage of the data is realized, the influence on the normal operation speed of the system is avoided, the preset type of target data of the buried point is obtained from the message queue, the secondary screening of the data is realized, the redundancy of the data is avoided, the target data of the buried point is further analyzed, the data table is generated, and the data table is stored in the configuration database, so that the asynchronous storage of the relational data is realized, the system operation pressure is avoided, the operation performance of the system is effectively improved, and the use of the subsequent data is facilitated because the relation among all data in the stored data is more definite.
Fig. 3 is a schematic structural diagram of an electronic device according to a preferred embodiment of the present invention.
In one embodiment of the present invention, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and a computer program, such as a data storage program, stored in the memory 12 and executable on the processor 13.
It will be appreciated by a person skilled in the art that the schematic diagram is only an example of the electronic device 1 and does not constitute a limitation of the electronic device 1, and that it may comprise more or less components than shown, or some components may be combined, or different components, e.g. the electronic device 1 may further comprise an input output device, a network access device, a bus, etc.
The Processor 13 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. The processor 13 is an operation core and a control center of the electronic device 1, and is connected to each part of the whole electronic device 1 by various interfaces and lines, and executes an operating system of the electronic device 1 and various installed application programs, program codes, and the like.
The processor 13 executes an operating system of the electronic device 1 and various installed application programs. The processor 13 executes the application program to implement the steps in the above-described respective data storage method embodiments, such as the steps S10, S11, S12, S13 shown in fig. 1.
Alternatively, the processor 13, when executing the computer program, implements the functions of the modules/units in the above device embodiments, for example: when data of buried points is received, writing the data of the buried points into a message queue; acquiring preset type target buried point data from the message queue; analyzing the target buried point data to generate a data table; storing the data table in a configuration database.
Illustratively, the computer program may be divided into one or more modules/units, which are stored in the memory 12 and executed by the processor 13 to accomplish the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a writing unit 110, an acquisition unit 111, a generation unit 112, a storage unit 113, a display unit 114, a determination unit 115, a calling unit 116, a transmission unit 117, and a detection unit 118.
The memory 12 can be used for storing the computer programs and/or modules, and the processor 13 implements various functions of the electronic device 1 by running or executing the computer programs and/or modules stored in the memory 12 and calling data stored in the memory 12. The memory 12 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data) created according to the use of the electronic device, and the like. Further, the memory 12 may include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other non-volatile solid state storage device.
The memory 12 may be an external memory and/or an internal memory of the electronic device 1. Further, the memory 12 may be a circuit with a memory function, such as a FIFO (First InFirst Out), which is not in physical form in an integrated circuit. Alternatively, the memory 12 may be a memory in a physical form, such as a memory stick, a TF Card (Trans-flash Card), or the like.
The integrated modules/units of the electronic device 1 may be stored in a computer-readable storage medium if they are implemented in the form of software functional units and sold or used as separate products. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments may be implemented.
Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
In conjunction with fig. 1, the memory 12 in the electronic device 1 stores a plurality of instructions to implement a data storage method, and the processor 13 can execute the plurality of instructions to implement: when a packaging instruction is received, a test page is obtained; when data of buried points is received, writing the data of the buried points into a message queue; acquiring preset type target buried point data from the message queue; analyzing the target buried point data to generate a data table; storing the data table in a configuration database.
Specifically, the processor 13 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1 for a specific implementation method of the instruction, which is not described herein again.
In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method of data storage, the method comprising:
when data of buried points is received, writing the data of the buried points into a message queue;
acquiring preset type target buried point data from the message queue;
analyzing the target buried point data to generate a data table;
storing the data table in a configuration database.
2. The data storage method of claim 1, wherein the method further comprises:
when a login signal is detected, acquiring all data operation types;
displaying all the data operation types;
when a selection signal of any operation type in all the data operation types is received, acquiring first data corresponding to the selected first operation type;
and determining the first data as the buried point data.
3. The data storage method of claim 1, wherein when the buried point data is received, the method further comprises:
calling a configuration interface;
connecting to a configuration server through the configuration interface;
and sending the buried point data to the configuration server.
4. The data storage method of claim 2, wherein the obtaining of the preset type of target buried point data from the message queue comprises one or more of the following combinations:
displaying all sub-operation types of the first operation type, when a selection signal of any operation type in all the sub-operation types is received, acquiring second data corresponding to the selected preset type, and determining the second data as the target buried point data; and/or
And determining the operation type of the mark as the preset type, acquiring second data corresponding to the preset type, and determining the second data as the target buried point data.
5. The data storage method of claim 1, wherein said parsing said target burial point data to generate a data table comprises:
and connecting at least one server, and analyzing the target buried point data by the at least one server to generate a data table.
6. The data storage method of claim 1, wherein said parsing said target burial point data to generate a data table further comprises:
analyzing the target buried point data by adopting a FastJson technology or a GSON technology to obtain the relation among all data in the target buried point data;
and establishing the data table based on the relation among all data in the target buried point data.
7. The data storage method of claim 1, wherein when storing the data table in a configuration database, the method further comprises:
detecting other data to be stored except the data table;
and when detecting that other data to be stored exist, storing the other data to be stored and the data table in the configuration database at the same time.
8. A data storage device, characterized in that the device comprises:
the writing unit is used for writing the buried point data into a message queue when the buried point data is received;
the acquisition unit is used for acquiring preset type target buried point data from the message queue;
the generating unit is used for analyzing the target buried point data to generate a data table;
and the storage unit is used for storing the data table in a configuration database.
9. An electronic device, characterized in that the electronic device comprises:
a memory storing at least one instruction; and
a processor executing instructions stored in the memory to implement a data storage method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium characterized by: the computer-readable storage medium has stored therein at least one instruction that is executed by a processor in an electronic device to implement the data storage method of any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910881337.0A CN110825731B (en) | 2019-09-18 | 2019-09-18 | Data storage method, device, electronic equipment and storage medium |
| PCT/CN2019/118140 WO2021051589A1 (en) | 2019-09-18 | 2019-11-13 | Data storage method and apparatus, electronic device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910881337.0A CN110825731B (en) | 2019-09-18 | 2019-09-18 | Data storage method, device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110825731A true CN110825731A (en) | 2020-02-21 |
| CN110825731B CN110825731B (en) | 2023-10-24 |
Family
ID=69547989
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910881337.0A Active CN110825731B (en) | 2019-09-18 | 2019-09-18 | Data storage method, device, electronic equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110825731B (en) |
| WO (1) | WO2021051589A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111523073A (en) * | 2020-04-23 | 2020-08-11 | 深圳市商汤科技有限公司 | Data management method and device, electronic equipment and storage medium |
| CN112416945A (en) * | 2020-12-07 | 2021-02-26 | 恩亿科(北京)数据科技有限公司 | Data processing method and system based on big data platform and computer equipment |
| CN114168430A (en) * | 2022-01-06 | 2022-03-11 | 携程旅游网络技术（上海）有限公司 | Front-end abnormal alarm configuration method, system, equipment and storage medium |
| CN115134352A (en) * | 2022-06-27 | 2022-09-30 | 重庆长安汽车股份有限公司 | Method, device, equipment and medium for uploading data of buried points |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11922026B2 (en) | 2022-02-16 | 2024-03-05 | T-Mobile Usa, Inc. | Preventing data loss in a filesystem by creating duplicates of data in parallel, such as charging data in a wireless telecommunications network |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107992506A (en) * | 2016-10-27 | 2018-05-04 | 北京京东尚科信息技术有限公司 | Information processing method and device |
| CN107995266A (en) * | 2017-11-22 | 2018-05-04 | 平安科技（深圳）有限公司 | Bury point data processing method, device, computer equipment and storage medium |
| CN108156006A (en) * | 2016-12-05 | 2018-06-12 | 阿里巴巴集团控股有限公司 | One kind buries point data report method, device and electronic equipment |
| CN108595450A (en) * | 2017-11-29 | 2018-09-28 | 五八有限公司 | One kind burying point data base construction method, bury point data base and buries a querying method |
| CN108874559A (en) * | 2018-05-31 | 2018-11-23 | 康键信息技术（深圳）有限公司 | electronic device, distributed system service link analysis method and storage medium |
| CN110060118A (en) * | 2019-02-27 | 2019-07-26 | 浙江执御信息技术有限公司 | A kind of order is honoured an agreement full link method for real-time monitoring, device and computer equipment |
| CN110147398A (en) * | 2019-04-25 | 2019-08-20 | 北京字节跳动网络技术有限公司 | A kind of data processing method, device, medium and electronic equipment |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106708899B (en) * | 2015-11-17 | 2021-04-27 | 阿里巴巴集团控股有限公司 | Automatic point burying method and device |
| CN106790718A (en) * | 2017-03-16 | 2017-05-31 | 北京搜狐新媒体信息技术有限公司 | Service call link analysis method and system |
| CN109582827A (en) * | 2018-11-30 | 2019-04-05 | 深圳市小牛普惠投资管理有限公司 | Nothing buries point data acquisition method, device, computer equipment and storage medium |
-
2019
- 2019-09-18 CN CN201910881337.0A patent/CN110825731B/en active Active
- 2019-11-13 WO PCT/CN2019/118140 patent/WO2021051589A1/en active Application Filing
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107992506A (en) * | 2016-10-27 | 2018-05-04 | 北京京东尚科信息技术有限公司 | Information processing method and device |
| CN108156006A (en) * | 2016-12-05 | 2018-06-12 | 阿里巴巴集团控股有限公司 | One kind buries point data report method, device and electronic equipment |
| CN107995266A (en) * | 2017-11-22 | 2018-05-04 | 平安科技（深圳）有限公司 | Bury point data processing method, device, computer equipment and storage medium |
| CN108595450A (en) * | 2017-11-29 | 2018-09-28 | 五八有限公司 | One kind burying point data base construction method, bury point data base and buries a querying method |
| CN108874559A (en) * | 2018-05-31 | 2018-11-23 | 康键信息技术（深圳）有限公司 | electronic device, distributed system service link analysis method and storage medium |
| CN110060118A (en) * | 2019-02-27 | 2019-07-26 | 浙江执御信息技术有限公司 | A kind of order is honoured an agreement full link method for real-time monitoring, device and computer equipment |
| CN110147398A (en) * | 2019-04-25 | 2019-08-20 | 北京字节跳动网络技术有限公司 | A kind of data processing method, device, medium and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| 崔伟;汪诗林;: "分布式系统中数据同步机制的研究与实现", 计算机工程与设计, no. 10, pages 2259 - 2261 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111523073A (en) * | 2020-04-23 | 2020-08-11 | 深圳市商汤科技有限公司 | Data management method and device, electronic equipment and storage medium |
| CN112416945A (en) * | 2020-12-07 | 2021-02-26 | 恩亿科(北京)数据科技有限公司 | Data processing method and system based on big data platform and computer equipment |
| CN114168430A (en) * | 2022-01-06 | 2022-03-11 | 携程旅游网络技术（上海）有限公司 | Front-end abnormal alarm configuration method, system, equipment and storage medium |
| CN115134352A (en) * | 2022-06-27 | 2022-09-30 | 重庆长安汽车股份有限公司 | Method, device, equipment and medium for uploading data of buried points |
| CN115134352B (en) * | 2022-06-27 | 2023-06-20 | 重庆长安汽车股份有限公司 | Buried point data uploading method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021051589A1 (en) | 2021-03-25 |
| CN110825731B (en) | 2023-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110825731B (en) | Data storage method, device, electronic equipment and storage medium | |
| US11755452B2 (en) | Log data collection method based on log data generated by container in application container environment, log data collection device, storage medium, and log data collection system | |
| CN110750592B (en) | Data synchronization method, device and terminal equipment | |
| CN112527848B (en) | Report data query method, device and system based on multiple data sources and storage medium | |
| CN111352800A (en) | Big data cluster monitoring method and related equipment | |
| CN111797351A (en) | Page data management method and device, electronic equipment and medium | |
| CN109800124B (en) | CPU utilization monitoring method and device, electronic equipment and storage medium | |
| CN113761514B (en) | Cloud desktop multi-factor security authentication method and system | |
| CN114116108A (en) | Dynamic rendering method, device, equipment and storage medium | |
| CN114169303A (en) | Method, device, equipment and medium for editing table based on vue.js | |
| CN114265657A (en) | Method and device for displaying page of applet | |
| CN112711398A (en) | Method, device and equipment for generating buried point file and storage medium | |
| CN113590593A (en) | Method and device for generating data table information, storage medium and electronic device | |
| CN115314570B (en) | Data issuing method, device, equipment and medium based on protocol development framework | |
| CN115794621A (en) | Code coverage test method, device, equipment and storage medium | |
| CN112817742B (en) | Data migration method, device, equipment and storage medium | |
| CN111741046B (en) | Data reporting method, data acquisition method, device, equipment and medium | |
| CN109828885B (en) | RocketMQ memory monitoring method and device, electronic equipment and storage medium | |
| CN111596897B (en) | Code multiplexing processing method and device and electronic equipment | |
| CN109840179B (en) | RocktMQ thread number monitoring method and device, electronic equipment and storage medium | |
| CN112181485A (en) | Script execution method and device, electronic equipment and storage medium | |
| CN108573042B (en) | Report synchronization method, electronic equipment and computer readable storage medium | |
| CN112667721A (en) | Data analysis method, device, equipment and storage medium | |
| CN110750569A (en) | Data extraction method, device, equipment and storage medium | |
| CN111859985A (en) | AI customer service model testing method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |