CN114153842A - Cross-platform data processing method, system, equipment and medium - Google Patents
Cross-platform data processing method, system, equipment and medium Download PDFInfo
- Publication number
- CN114153842A CN114153842A CN202111340097.7A CN202111340097A CN114153842A CN 114153842 A CN114153842 A CN 114153842A CN 202111340097 A CN202111340097 A CN 202111340097A CN 114153842 A CN114153842 A CN 114153842A
- Authority
- CN
- China
- Prior art keywords
- data
- access
- main
- platform
- cross
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention discloses a cross-platform data processing method, a system, equipment and a medium, wherein the method comprises the following steps: analyzing and processing the logical relation of the system data table according to the platform service flow, and determining a data index dictionary; identifying the main data according to the data index dictionary, storing the main data through a distributed storage technology, and determining a main data copy; and when data is accessed in a cross-platform mode, data access is carried out on the data index dictionary and the main data copy through data pre-reading. According to the invention, the main data is stored in a distributed manner, so that the storage efficiency of the main data is improved, the access efficiency of the data is improved through data pre-reading, and the method can be widely applied to the technical field of data processing.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a cross-platform data processing method, a cross-platform data processing system, cross-platform data processing equipment and a cross-platform data processing medium.
Background
The digital economy is supplied by rich digital elements, the element configuration efficiency is improved in a networking mode, the output efficiency is intelligently improved, and the quality change, the efficiency change and the power change of the economy development are promoted. Wherein, the data information resource gradually becomes a new key element resource. With the rapid development of the information technology, with the appearance of various platforms, data types become diversified gradually, data sizes become larger and larger, and in the prior art, a plurality of systems need to be reconstructed, but system reconstruction not only affects the operation of normal services, but also has low access efficiency to system data.
Disclosure of Invention
In view of this, embodiments of the present invention provide a cross-platform data processing method, system, device and medium, so as to improve system access and storage efficiency and reduce data reading time.
In one aspect, the present invention provides a cross-platform data processing method, including:
analyzing and processing the logical relation of the system data table according to the platform service flow, and determining a data index dictionary;
identifying the main data according to the data index dictionary, storing the main data through a distributed storage technology, and determining a main data copy;
and when data is accessed in a cross-platform mode, data access is carried out on the data index dictionary and the main data copy through data pre-reading.
Optionally, the analyzing and processing the logical relationship of the system data table according to the platform service flow to determine the data index dictionary includes:
recording the sequence of data generation in a system data table according to the platform service flow, and determining source data;
inquiring the dependency relationship of the source data according to the business process to determine the dependency data;
and adding indexes to the source data, and determining a data index dictionary by combining the dependent data.
Optionally, the identifying the master data according to the data index dictionary, storing the master data through a distributed storage technology, and determining a master data copy includes:
identifying the business entity attribute in the data index dictionary to determine main data;
and recording the main data into a cache database, and performing vertical segmentation processing and horizontal segmentation processing on the cache database to determine a main data copy.
Optionally, the recording the main data into a cache database, performing vertical segmentation processing and horizontal segmentation processing on the cache database, and determining a main data copy includes:
recording the main data into a cache database, performing vertical segmentation processing on the cache database, classifying cache data tables in the cache database according to different service types, and storing the cache data tables into a first database;
performing horizontal segmentation processing on the first database, and storing data rows of a cache data table in the first database into a second database according to Hash modulo division;
and determining a primary data copy according to the first database and the second database.
Optionally, when accessing data across platforms, performing data access on the data index dictionary and the primary data copy through data pre-reading includes:
when data are accessed in a cross-platform mode, judging that the data access is service access data, and performing data access on the service data in the data index dictionary through data pre-reading;
when data are accessed in a cross-platform mode, the data access is judged to be main data access, and data access is conducted on the main data in the main data copy through data pre-reading.
Optionally, when accessing data across platforms, determining that the data access is service data access, and performing data access on the service data in the data index dictionary through data pre-reading includes:
when data are accessed in a cross-platform mode, judging that the data access is access service data, pre-reading the data according to the data access probability in the data index dictionary, and determining pre-read data;
when the pre-read data is target access data, accessing the pre-read data;
and when the pre-read data is not target access data, positioning the target access data through the index of the data index dictionary, and performing data access.
Optionally, when accessing data across platforms, determining that the data access is the main data access, and performing data access on the main data in the main data copy through data pre-reading includes:
when data are accessed in a cross-platform mode, judging that the data access is main access data, pre-reading the data according to the data access probability in the main data copy, and determining pre-read data;
when the pre-read data is target access data, accessing the pre-read data;
and when the pre-read data is not the target access data, searching the target access data through the main data copy to access the data.
On the other hand, the embodiment of the invention also discloses a cross-platform data processing system, which comprises:
the first module is used for analyzing and processing the logical relation of the system data table according to the platform service flow and determining a data index dictionary;
the second module is used for identifying the main data according to the data index dictionary, storing the main data through a distributed storage technology and determining a main data copy;
and the third module is used for performing data access on the data index dictionary and the main data copy through data pre-reading when data are accessed in a cross-platform mode.
On the other hand, the embodiment of the invention also discloses an electronic device, which comprises a processor and a memory;
the memory is used for storing programs;
the processor executes the program to implement the method as described above.
On the other hand, the embodiment of the invention also discloses a computer readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to realize the method.
In another aspect, an embodiment of the present invention further discloses a computer program product or a computer program, where the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The computer instructions may be read by a processor of a computer device from a computer-readable storage medium, and the computer instructions executed by the processor cause the computer device to perform the foregoing method.
Compared with the prior art, the invention adopting the technical scheme has the following technical effects: according to the platform business process, the logical relation of a system data table is analyzed and processed, and a data index dictionary is determined; identifying the main data according to the data index dictionary, storing the main data through a distributed storage technology, determining a main data copy, and improving the storage efficiency of the main data by establishing the distributed main data copy; when data are accessed in a cross-platform mode, data access is conducted on the data index dictionary and the main data copy through data pre-reading, and data access efficiency can be improved through data pre-reading.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart of a cross-platform data processing method according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
Referring to fig. 1, an embodiment of the present invention provides a method, including:
analyzing and processing the logical relation of the system data table according to the platform service flow, and determining a data index dictionary;
identifying the main data according to the data index dictionary, storing the main data through a distributed storage technology, and determining a main data copy;
and when data is accessed in a cross-platform mode, data access is carried out on the data index dictionary and the main data copy through data pre-reading.
The data indexing method comprises the steps of merging various data sources in a system data table according to different business processes, such as retail business, customer service business and the like, positioning, classifying and analyzing data through visual configuration according to the logical relation between the business processes and the data sources, and adding an extensible data indexing system to form a data indexing dictionary. By identifying the main data in the data index dictionary, the main data are stored in the main data copy in a distributed manner, so that the storage efficiency of the main data is improved. It should be noted that, in the embodiment of the present invention, data is classified into main data and service data, the main data is bottom-layer basic data in the system and is used for defining a service object, the service data is data generated according to a service flow through the main data, for example, the main data in retail industry is entity data of manufacturers, goods, and the like, and the service data is service data of one hundred tons of goods produced by the manufacturers and the like and is used for recording the service flow of the main data. When data are accessed in a cross-platform mode, the data are classified, analyzed and mined through the main data copy and the data index dictionary, the association relation between the data tables is analyzed, the characteristic attributes of the user are obtained, the user is predicted according to the characteristic attributes, the data object set to be accessed by the user program is predicted, and the data objects are read in advance, so that the time consumption is reduced.
Further as a preferred embodiment, the analyzing and processing the logical relationship of the system data table according to the platform service process to determine the data index dictionary includes:
recording the sequence of data generation in a system data table according to the platform service flow, and determining source data;
inquiring the dependency relationship of the source data according to the business process to determine the dependency data;
and adding indexes to the source data, and determining a data index dictionary by combining the dependent data.
And recording the data according to the sequence of the generation of the business link data of different platforms to obtain the source data. And searching to obtain the dependent data according to the dependency relationship among the data. The dependency relationship among the data can be obtained by specifically analyzing the business process, indexes are added to the source data, the dependency data can be obtained by searching according to the source data, and a data index dictionary is established. In this embodiment, starting from the viewpoint of not changing the existing data islands, source data are determined according to the sequence of the generation of the business link data by establishing a data intermediate layer, and an index is described and established in the data intermediate layer to clear up the relationship between the data; the method lays a foundation for flow scheduling and data analysis, and can solve the problem of data consistency.
Further as a preferred embodiment, the identifying the master data according to the data index dictionary, storing the master data through a distributed storage technology, and determining a master data copy includes:
identifying the business entity attribute in the data index dictionary to determine main data;
and recording the main data into a cache database, and performing vertical segmentation processing and horizontal segmentation processing on the cache database to determine a main data copy.
The main data is the bottom basic data in the system and is used for defining the data of the business object, and the main data in the data index dictionary can be identified according to the attributes of the business entity. The main data is recorded in the cache database, the data in the cache database is distributed into a plurality of different databases according to a certain rule through distributed storage, and the request is distributed into the plurality of different databases, so that the access pressure of the databases is well relieved, and the overall availability of the system can be improved while data segmentation is carried out. And aiming at the main data which needs to be frequently read, a cache database is established in the data middle layer, and the cache main data is stored in a distributed manner by adopting a horizontal segmentation mode and a vertical segmentation mode, so that the access data magnitude is reduced. The vertical segmentation and the horizontal segmentation have advantages and disadvantages, and the embodiment adopts a distributed system architecture mode combining the horizontal segmentation and the vertical segmentation, so that the access efficiency of the system is improved, and the reliability of the system is improved. Tables with closely related services are combined together and divided into a database, and under the condition of reducing the coupling among various services, data of one table is distributed into a plurality of small tables through hashing, so that the table query efficiency is improved, and the database reading efficiency is improved; the access pressure of the centralized database is effectively relieved, and the performance of the database is improved.
Further as a preferred embodiment, the recording the main data into a cache database, performing vertical slicing processing and horizontal slicing processing on the cache database, and determining a main data copy includes:
recording the main data into a cache database, performing vertical segmentation processing on the cache database, classifying cache data tables in the cache database according to different service types, and storing the cache data tables into a first database;
performing horizontal segmentation processing on the first database, and storing data rows of a cache data table in the first database into a second database according to Hash modulo division;
and determining a primary data copy according to the first database and the second database.
The main data is recorded into a cache database, the cache database is composed of a plurality of tables, each table corresponds to different services, the cache data tables in the cache database are classified according to different service types and are vertically divided into different first databases. And further horizontally dividing the classified cache data table in the first database, and storing the data rows of the cache data table in the first database into a second database according to a Hash modulo mode. All of the first database and the second database are determined to be primary data replicas.
Further as a preferred embodiment, when accessing data across platforms, performing data access on the data index dictionary and the master data copy through data pre-reading includes:
when data are accessed in a cross-platform mode, judging that the data access is service access data, and performing data access on the service data in the data index dictionary through data pre-reading;
when data are accessed in a cross-platform mode, the data access is judged to be main data access, and data access is conducted on the main data in the main data copy through data pre-reading.
When data are accessed in a cross-platform mode, the type of the data access is judged, and if the data access is the access service data, the data access is carried out on the service data in the data index dictionary through data pre-reading; and if the data access is the main data access, performing data access on the main data in the main data copy through data pre-reading. In the embodiment, the business data is searched through the data index dictionary, the corresponding business data can be positioned through searching the keyword index, and the data query speed can be improved by searching the business data through the data index dictionary; the main data is searched through the main data copy, and the main data copy stores the main data through a distributed storage technology, so that the access efficiency of the main data can be improved.
Further as a preferred embodiment, when accessing data across platforms, determining that the data access is service data access, and performing data access on the service data in the data index dictionary through data pre-reading includes:
when data are accessed in a cross-platform mode, judging that the data access is access service data, pre-reading the data according to the data access probability in the data index dictionary, and determining pre-read data;
when the pre-read data is target access data, accessing the pre-read data;
and when the pre-read data is not target access data, positioning the target access data through the index of the data index dictionary, and performing data access.
After the data index dictionary is generated, the incidence relation of the data is integrated through the indexes, and the access frequency of the data is dynamically calculated according to the attribute, the service category and the like of the data, so that the access frequency of each data is obtained. When the cross-platform access service data is accessed, the data is pre-read according to the service attribute of the target access data, the data with higher access probability is determined as pre-read data and is placed in the buffer area, and when the pre-read data is the target access data, the data is accessed by accessing the pre-read data in the buffer area, so that the data reading efficiency is improved. And when the pre-read data is not the target access data, positioning the target access data through the indexes in the data index dictionary for data access.
Further as a preferred embodiment, when accessing data across platforms, determining that the data access is the main data access, and performing data access on the main data in the main data copy through data pre-reading includes:
when data are accessed in a cross-platform mode, judging that the data access is main access data, pre-reading the data according to the data access probability in the main data copy, and determining pre-read data;
when the pre-read data is target access data, accessing the pre-read data;
and when the pre-read data is not the target access data, searching the target access data through the main data copy to access the data.
And dynamically calculating the data access probability of the data in the main data copy to obtain the access probability of each data. When main data is accessed, the data is pre-read according to the attribute of the main data, the data with higher access probability is determined as pre-read data and is placed in the buffer area, and when the pre-read data is target access data, the data is accessed through accessing the pre-read data in the buffer area, so that the data reading efficiency is improved. And when the pre-read data is not the target access data, querying the primary data copy to locate the target access data for data access. According to the embodiment, the frequent access data of the user can be analyzed according to the access condition of the user, so that the performance of the system is improved, and personalized service is provided for the user. By predicting the data of the user access, the data which can be accessed by the user in the next step or in the future can be predicted according to the historical access data and the current access data of the user. The predicted result is sent to the user in advance, and when the user accesses the data, the data can be read only by a local cache, so that the access efficiency of the user and the utilization rate of communication resources are improved.
The embodiment of the invention also discloses a cross-platform data processing system, which comprises:
the first module is used for analyzing and processing the logical relation of the system data table according to the platform service flow and determining a data index dictionary;
the second module is used for identifying the main data according to the data index dictionary, storing the main data through a distributed storage technology and determining a main data copy;
and the third module is used for performing data access on the data index dictionary and the main data copy through data pre-reading when data are accessed in a cross-platform mode.
Corresponding to the method of fig. 1, an embodiment of the present invention further provides an electronic device, including a processor and a memory; the memory is used for storing programs; the processor executes the program to implement the method as described above.
Corresponding to the method of fig. 1, the embodiment of the present invention also provides a computer-readable storage medium, which stores a program, and the program is executed by a processor to implement the method as described above.
The embodiment of the invention also discloses a computer program product or a computer program, which comprises computer instructions, and the computer instructions are stored in a computer readable storage medium. The computer instructions may be read by a processor of a computer device from a computer-readable storage medium, and executed by the processor to cause the computer device to perform the method illustrated in fig. 1.
In summary, the embodiments of the present invention have the following advantages:
(1) according to the embodiment of the invention, the data index dictionary is established, and the relationship between the data is combed, so that the access efficiency and the utilization rate of the data can be improved;
(2) according to the embodiment of the invention, the distributed primary data copy is established, so that the storage efficiency of primary data can be improved;
(3) according to the embodiment of the invention, the data access is carried out on the data index dictionary and the main data copy through data pre-reading, so that the data access efficiency can be improved.
In alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flow charts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are performed independently.
Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and/or features may be integrated in a single physical device and/or software module, or one or more functions and/or features may be implemented in a separate physical device or software module. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary for an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be understood within the ordinary skill of an engineer, given the nature, function, and internal relationship of the modules. Accordingly, those skilled in the art can, using ordinary skill, practice the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative of and not intended to limit the scope of the invention, which is defined by the appended claims and their full scope of equivalents.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
While the preferred embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A cross-platform data processing method is characterized by comprising the following steps:
analyzing and processing the logical relation of the system data table according to the platform service flow, and determining a data index dictionary;
identifying the main data according to the data index dictionary, storing the main data through a distributed storage technology, and determining a main data copy;
and when data is accessed in a cross-platform mode, data access is carried out on the data index dictionary and the main data copy through data pre-reading.
2. The method according to claim 1, wherein the analyzing and processing the logical relationship of the system data table according to the platform business process to determine the data index dictionary comprises:
recording the sequence of data generation in a system data table according to the platform service flow, and determining source data;
inquiring the dependency relationship of the source data according to the business process to determine the dependency data;
and adding indexes to the source data, and determining a data index dictionary by combining the dependent data.
3. The cross-platform data processing method according to claim 1, wherein the identifying the master data according to the data index dictionary, storing the master data through a distributed storage technology, and determining the master data copy comprises:
identifying the business entity attribute in the data index dictionary to determine main data;
and recording the main data into a cache database, and performing vertical segmentation processing and horizontal segmentation processing on the cache database to determine a main data copy.
4. The cross-platform data processing method according to claim 3, wherein the recording the main data into a cache database, performing vertical slicing processing and horizontal slicing processing on the cache database, and determining a main data copy comprises:
recording the main data into a cache database, performing vertical segmentation processing on the cache database, classifying cache data tables in the cache database according to different service types, and storing the cache data tables into a first database;
performing horizontal segmentation processing on the first database, and storing data rows of a cache data table in the first database into a second database according to Hash modulo division;
and determining a primary data copy according to the first database and the second database.
5. The cross-platform data processing method according to claim 1, wherein when accessing data in a cross-platform manner, performing data access on the data index dictionary and the master data copy through data pre-reading comprises:
when data are accessed in a cross-platform mode, judging that the data access is service access data, and performing data access on the service data in the data index dictionary through data pre-reading;
when data are accessed in a cross-platform mode, the data access is judged to be main data access, and data access is conducted on the main data in the main data copy through data pre-reading.
6. The cross-platform data processing method according to claim 5, wherein when accessing data in a cross-platform manner, determining that the data access is service data access, and performing data access on the service data in the data index dictionary through data pre-reading comprises:
when data are accessed in a cross-platform mode, judging that the data access is access service data, pre-reading the data according to the data access probability in the data index dictionary, and determining pre-read data;
when the pre-read data is target access data, accessing the pre-read data;
and when the pre-read data is not target access data, positioning the target access data through the index of the data index dictionary, and performing data access.
7. The cross-platform data processing method according to claim 5, wherein when accessing data in a cross-platform manner, determining that the data access is the main data access, and performing data access on the main data in the main data copy through data pre-reading comprises:
when data are accessed in a cross-platform mode, judging that the data access is main access data, pre-reading the data according to the data access probability in the main data copy, and determining pre-read data;
when the pre-read data is target access data, accessing the pre-read data;
and when the pre-read data is not the target access data, searching the target access data through the main data copy to access the data.
8. A cross-platform data processing system, comprising:
the first module is used for analyzing and processing the logical relation of the system data table according to the platform service flow and determining a data index dictionary;
the second module is used for identifying the main data according to the data index dictionary, storing the main data through a distributed storage technology and determining a main data copy;
and the third module is used for performing data access on the data index dictionary and the main data copy through data pre-reading when data are accessed in a cross-platform mode.
9. An electronic device comprising a processor and a memory;
the memory is used for storing programs;
the processor executing the program realizes the method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that the storage medium stores a program, which is executed by a processor to implement the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111340097.7A CN114153842B (en) | 2021-11-12 | 2021-11-12 | Cross-platform data processing method, system, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111340097.7A CN114153842B (en) | 2021-11-12 | 2021-11-12 | Cross-platform data processing method, system, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114153842A true CN114153842A (en) | 2022-03-08 |
| CN114153842B CN114153842B (en) | 2022-05-20 |
Family
ID=80460259
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111340097.7A Active CN114153842B (en) | 2021-11-12 | 2021-11-12 | Cross-platform data processing method, system, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114153842B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064762A (en) * | 2012-12-25 | 2013-04-24 | 华为技术有限公司 | Method and device for recovering deleted backup data |
| CN103729455A (en) * | 2014-01-06 | 2014-04-16 | 中国南方电网有限责任公司 | Master data storage method based on primary copy storage pattern |
| CN104516967A (en) * | 2014-12-25 | 2015-04-15 | 国家电网公司 | Electric power system mass data management system and use method thereof |
| US20160342661A1 (en) * | 2015-05-20 | 2016-11-24 | Commvault Systems, Inc. | Handling user queries against production and archive storage systems, such as for enterprise customers having large and/or numerous files |
| CN108363764A (en) * | 2018-02-05 | 2018-08-03 | 山东地主网络科技创新有限公司 | A kind of distributed caching management system and method |
| CN109783548A (en) * | 2018-11-26 | 2019-05-21 | 远光软件股份有限公司 | Master data fusion visualization display methods, system, electronic device and storage medium |
| CN111241088A (en) * | 2018-11-09 | 2020-06-05 | 北京京东尚科信息技术有限公司 | Data writing method, data query method, device and equipment |
| CN112286903A (en) * | 2020-09-27 | 2021-01-29 | 苏州浪潮智能科技有限公司 | Containerization-based relational database optimization method and device |
| CN112800019A (en) * | 2021-03-03 | 2021-05-14 | 国网甘肃省电力公司 | Data backup method and system based on Hadoop distributed file system |
| CN113282589A (en) * | 2021-06-17 | 2021-08-20 | 中国建设银行股份有限公司 | Data acquisition method and device |
-
2021
- 2021-11-12 CN CN202111340097.7A patent/CN114153842B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064762A (en) * | 2012-12-25 | 2013-04-24 | 华为技术有限公司 | Method and device for recovering deleted backup data |
| CN103729455A (en) * | 2014-01-06 | 2014-04-16 | 中国南方电网有限责任公司 | Master data storage method based on primary copy storage pattern |
| CN104516967A (en) * | 2014-12-25 | 2015-04-15 | 国家电网公司 | Electric power system mass data management system and use method thereof |
| US20160342661A1 (en) * | 2015-05-20 | 2016-11-24 | Commvault Systems, Inc. | Handling user queries against production and archive storage systems, such as for enterprise customers having large and/or numerous files |
| CN108363764A (en) * | 2018-02-05 | 2018-08-03 | 山东地主网络科技创新有限公司 | A kind of distributed caching management system and method |
| CN111241088A (en) * | 2018-11-09 | 2020-06-05 | 北京京东尚科信息技术有限公司 | Data writing method, data query method, device and equipment |
| CN109783548A (en) * | 2018-11-26 | 2019-05-21 | 远光软件股份有限公司 | Master data fusion visualization display methods, system, electronic device and storage medium |
| CN112286903A (en) * | 2020-09-27 | 2021-01-29 | 苏州浪潮智能科技有限公司 | Containerization-based relational database optimization method and device |
| CN112800019A (en) * | 2021-03-03 | 2021-05-14 | 国网甘肃省电力公司 | Data backup method and system based on Hadoop distributed file system |
| CN113282589A (en) * | 2021-06-17 | 2021-08-20 | 中国建设银行股份有限公司 | Data acquisition method and device |
Non-Patent Citations (4)
| Title |
|---|
| GUANGPING XU ET AL.: "LIPA: A Learning-based Indexing and Prefetching Approach for Data Deduplication", 《2019 35TH SYMPOSIUM ON MASS STORAGE SYSTEMS AND TECHNOLOGIES》 * |
| SHIMIN CHEN ET AL.: "Improving Index Performance through Prefetching", 《ACM SIGMOD RECORD》 * |
| 张鹏: "基于主数据的物料编码和收集过程管控的研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 王晓晋: "基于组件的大型商贸企业主数据管理系统架构设计", 《铁路采购与物流》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114153842B (en) | 2022-05-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019218B (en) | Data storage and query method and equipment | |
| US11132388B2 (en) | Efficient spatial queries in large data tables | |
| US10977248B2 (en) | Processing records in dynamic ranges | |
| US8874600B2 (en) | System and method for building a cloud aware massive data analytics solution background | |
| US20110179013A1 (en) | Search Log Online Analytic Processing | |
| CN115905630A (en) | Graph database query method, device, equipment and storage medium | |
| CN105095515A (en) | Bucket dividing method, device and equipment supporting fast query of Map-Reduce output result | |
| CN112241396B (en) | Spark-based method and system for merging small files of Delta | |
| CN114153842B (en) | Cross-platform data processing method, system, equipment and medium | |
| CN107430633B (en) | System and method for data storage and computer readable medium | |
| US9576004B1 (en) | Free space management in databases | |
| CN103902614A (en) | Data processing method, device and system | |
| US20180349372A1 (en) | Media item recommendations based on social relationships | |
| US11645283B2 (en) | Predictive query processing | |
| CN113625967B (en) | Data storage method, data query method and server | |
| WO2022001626A1 (en) | Time series data injection method, time series data query method and database system | |
| US11567906B2 (en) | Generation and traversal of a hierarchical index structure for efficient data retrieval | |
| CN115269654A (en) | Data cache supplementing method, device, equipment and medium | |
| Jiadi et al. | Research on Data Center Operation and Maintenance Management Based on Big Data | |
| CN114297236A (en) | Data blood relationship analysis method, terminal equipment and storage medium | |
| CN113297267A (en) | Data caching and task processing method, device, equipment and storage medium | |
| CN111309704A (en) | Database operation method and database operation system | |
| CN110019448A (en) | A kind of data interactive method and device | |
| US20220413727A1 (en) | Quality-performance optimized identification of duplicate data | |
| KR20030065860A (en) | Video Searching Apparatus and its Method using XML Hierarchy Structure |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |