CN111460024B - Real-time service system based on elastic search - Google Patents
Real-time service system based on elastic search Download PDFInfo
- Publication number
- CN111460024B CN111460024B CN202010358051.7A CN202010358051A CN111460024B CN 111460024 B CN111460024 B CN 111460024B CN 202010358051 A CN202010358051 A CN 202010358051A CN 111460024 B CN111460024 B CN 111460024B
- Authority
- CN
- China
- Prior art keywords
- data
- node
- service data
- module
- index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24532—Query optimisation of parallel queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/083—Shipping
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Economics (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Entrepreneurship & Innovation (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Development Economics (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of logistics, provides an elastic search-based real-time service system, and aims to solve the problem that the management and control performance of an existing system frame on real-time service data is low. The real-time business system based on the elastic search comprises an application sending device, a data management device, a database synchronization device and a search engine device; the data management device is used for carrying out updating processing on the stored real-time service data according to the service data updating request sent by the application sending device to obtain updated service data, wherein the service data is logistics service data; the database synchronization device is used for synchronizing the update service data to the search engine device through an open source data synchronization tool Canal; the search engine device is used for carrying out separation processing and index creation processing on the updated service data through the search and data analysis engine elastic search to obtain the service data to be queried, and inquiring the service data to be queried according to the service data inquiry request sent by the application sending device to obtain target inquiry data.
Description
Technical Field
The invention relates to the technical field of logistics, in particular to a real-time service system based on an elastic search.
Background
With the continuous development of computer technology and the continuous improvement of informatization degree, a large amount of service data is generated, in order to manage the service data, the logistics industry uses a plurality of management type service systems in respective systems, and the service systems have a large amount of service object processing logic phenomena, for example: the operations of inserting, modifying and deleting the logistics business data are performed, and the logistics business data are inquired through multiple conditions, and the life cycle of the business data is controlled. The existing system frame is used for processing logic phenomena of a large number of business objects, and the system frame with a structured database is generally adopted for storing business data.
In the prior art, as the logistics traffic increases, the generated logistics traffic data is larger and larger, when the data size of the logistics traffic data exceeds millions, the single-table structure based on database persistence in the existing system frame cannot meet the performance requirement, and the sub-table sub-library structure in the existing system frame cannot meet the multi-condition query requirement, so that the management and control performance of the existing system frame on the real-time traffic data is low.
Disclosure of Invention
The invention mainly aims to solve the problem that the existing system framework has low management and control performance on real-time service data.
The invention provides a real-time business system based on an elastic search, which comprises an application sending device, a data management device, a database synchronization device and a search engine device, wherein the application sending device is used for sending an application request to the database synchronization device; wherein:
the application sending device is used for sending a service data updating request to the data management device and sending a service data query request to the search engine device;
the data management device is used for carrying out updating processing on the stored real-time service data according to the service data updating request to obtain updated service data, and sending the updated service data to the database synchronization device, wherein the service data is logistics service data;
the database synchronization device is used for synchronizing the update service data to the search engine device through a preset open source data synchronization tool Canal;
the search engine device is used for storing the updated service data through a preset search and data analysis engine, performing separation processing and index creation processing on the updated service data to obtain service data to be queried, and querying the service data to be queried according to the service data query request to obtain target query data.
Optionally, the search engine device comprises a search and data analysis engine elastsearch cluster, and the search and data analysis engine elastsearch cluster comprises a main network node server and a plurality of controlled network node servers;
the master network node server is respectively in communication connection with the plurality of controlled network node servers and is used for controlling the plurality of controlled network node servers to perform separation processing and index creation processing on the updated service data to obtain service data to be queried, and controlling the plurality of controlled network node servers to query the service data to be queried according to the service data query request to obtain target query data.
Optionally, the master network node server includes a processing module;
the processing module is used for marking hot nodes and cold nodes of the controlled network node servers according to preset service requirements, performing role node configuration on the controlled network node servers according to preset role configuration information, and performing index creation on the updated service data;
the processing module comprises a cold and hot separation sub-module, an index creation sub-module, a configuration sub-module, a role separation sub-module and a sending sub-module;
The cold-hot separation sub-module is configured to perform node marking on the plurality of controlled network node servers according to a preset time node to obtain a hot node and a cold node, classify the updated service data according to the hot node and the cold node to obtain hot node data and cold node data, and send the hot node data and the cold node data to the index creation sub-module, where the hot node corresponds to a preset target time period, the cold node corresponds to a time period outside the target time period, and the target time period is a query time period determined according to a preset service requirement;
the index creation sub-module is used for carrying out index creation on the hot node data and the cold node data to obtain node data with created indexes, and sending the node data with created indexes to the sending sub-module;
the configuration sub-module is used for storing configuration information of each type of the search engine device, wherein the configuration information comprises role configuration information and sends the role configuration information to the role separation sub-module;
the role separation module is configured to perform role node configuration on the plurality of controlled network node servers according to the role configuration information to obtain a plurality of configured controlled network node servers, wherein the role configuration information comprises a role, a node corresponding to the role and node configuration information of the node;
The sending submodule is used for sending the updated service data to the corresponding controlled network node server according to the hot node and the cold node, and sending the node data with the created index to the configured controlled network node server corresponding to the role according to the role and the node configuration information, so as to obtain the service data to be queried.
Optionally, the index creation submodule includes a first determining unit, a first creating unit, a sending unit, a second determining unit, a migration unit and a third determining unit;
the first determining unit is configured to determine whether the updated service data is within a preset service data life cycle, determine the updated service data as historical data if the updated service data is not within the preset service data life cycle, and send the historical data to the sending unit;
the first creating unit is configured to create an index cluster on the hot node, obtain corresponding first node data, and send the first node data to the second determining unit and the third determining unit, where the first node data includes a node index and a generation time of service data;
The sending unit is used for sending the historical data to the cold node to obtain target cold node data, and sending the target cold node data to the migration unit;
the second determining unit is configured to obtain a current time, determine a node index to be migrated in the node indexes according to the generation time and the current time, and send the node index to be migrated to the migration unit;
the migration unit is used for migrating the node index to be migrated to the cold node corresponding to the target cold node data through a preset interface to obtain corresponding second node data;
the third determining unit is configured to determine the first node data obtained by the first creating unit and the second node data obtained by the migration unit as node data having created an index.
Optionally, the index creating sub-module includes a second creating unit, a first setting unit, an analyzing unit, a second setting unit, and a fourth determining unit;
the second creating unit is configured to create an index for the hot node data and the cold node data, obtain an initial index, and send the initial index to the first setting unit;
The first setting unit is used for performing slicing setting on the initial index according to a preset main slicing number to obtain an initial index after slicing, and sending the initial index after slicing to the analysis unit;
the analysis unit is used for carrying out search requirement analysis, aggregation analysis and word segmentation requirement analysis on the segmented initial indexes to obtain candidate indexes, and sending the candidate indexes to the second setting unit;
the second setting unit is configured to perform attribute setting and type setting on the candidate indexes to obtain target indexes, and send the target indexes to the fourth determining unit;
the fourth determining unit is configured to determine hot node data created with the target index and cold node data created with the target index as node data of the created index.
Optionally, the master network node server further includes a query module;
the query module is used for querying the service data to be queried in the plurality of controlled network node servers according to the service data query request to obtain target query data;
the query module comprises an extraction submodule, an analysis submodule and a retrieval submodule;
The extraction sub-module is used for receiving the service data query request sent by the application sending device, extracting the query time information in the service data query request, and sending the query time information to the analysis sub-module;
the analysis submodule is used for carrying out node analysis on the plurality of configured controlled network node servers obtained by the processing module according to the inquiry time information and the role configuration information to obtain corresponding target nodes;
and the retrieval submodule is used for retrieving node data with created indexes in the target nodes obtained by the analysis submodule according to the service data query request to obtain target query data corresponding to the service data query request.
Optionally, the second determining unit includes an acquiring subunit, a calculating subunit, a judging subunit and a determining subunit;
the obtaining subunit is used for obtaining the current time and sending the current time to the calculating subunit;
the calculating subunit is used for calculating a time length difference value between the generating time and the current time and sending the time length difference value to the judging subunit;
The judging subunit is configured to judge whether the duration difference is greater than a preset threshold, obtain a judging result, and send the judging result to the determining subunit;
and the determining subunit is configured to determine, as the node index to be migrated, the node index corresponding to the first node data whose determination result is yes.
Optionally, the database synchronization device comprises a monitoring module, a cache module and a synchronization module;
the monitoring module is used for analyzing the incremental log of the database in the data management device in real time, extracting corresponding updating service data and sending the updating service data to the cache module;
the buffer module is used for buffering the update service data to obtain buffered update service data and sending the buffered update service data to the synchronization module;
and the synchronization module is used for synchronizing the cached update service data to the search engine device according to the synchronization configuration file in the open source data synchronization tool Canal.
Optionally, the real-time service system based on the elastic search further includes an optimization module, where the optimization module is configured to record and statistically analyze the service data query request and the target query data to obtain statistical analysis data, and optimize cold-hot separation of updated service data in the search engine device according to the statistical analysis data.
Optionally, the search engine device further includes a parameter setting module, where the parameter setting module is configured to set parameters in the search engine device, where the parameters include a refresh time interval, index fragment data, a number of fragments on a single node, a storage location and a storage manner of a transaction log, and an index setting policy.
The invention provides a real-time business system based on an elastic search, which meets the requirements on storage and running performance of a large amount of logistics business data and multi-condition query through a data management device, a database synchronization device and a search engine device, realizes synchronization and read-write separation of the logistics business data, and ensures that the storage and query modes of the real-time data can be selected in a single-table mode or a multi-table mode through the processing of the real-time data by the search engine device based on the elastic search, so that the system architecture has higher flexibility, thereby solving the problem of low management and control performance of the real-time business data by the existing system framework.
Drawings
FIG. 1 is a schematic block diagram of a real-time service system based on an elastic search in an embodiment of the present invention;
FIG. 2 is a schematic block diagram of a database synchronization apparatus according to an embodiment of the present invention;
FIG. 3 is a schematic block diagram of a search engine apparatus according to an embodiment of the present invention;
FIG. 4 is a schematic block diagram of an index creation sub-module in accordance with an embodiment of the present invention;
FIG. 5 is a block diagram schematically showing the structure of the second determining unit in the embodiment of the present invention
FIG. 6 is another block diagram of an index creation sub-module in accordance with an embodiment of the present invention;
FIG. 7 is a schematic block diagram of an analysis sub-module in accordance with an embodiment of the present invention;
fig. 8 is a schematic block diagram of another architecture of an elastic search-based real-time service system according to an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a real-time service system based on an elastic search, which solves the problem of low management and control performance of the existing system framework on real-time service data.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a module, unit, or system is not necessarily limited to those modules or units that are expressly listed or inherent to such system, but may include other modules or units that are not expressly listed or inherent to such system.
For easy understanding, the following describes a specific flow of an embodiment of the present invention, referring to fig. 1, a real-time service system based on elastic search in the embodiment of the present invention includes an application sending device 1, a data management device 2, a database synchronization device 3, and a search engine device 4; wherein:
an application transmitting means 1 for transmitting a service data update request to the data management means 2 and a service data query request to the search engine means 4;
the data management device 2 is configured to update the stored real-time service data according to the service data update request to obtain updated service data, and send the updated service data to the database synchronization device 3, where the service data is logistics service data;
a database synchronization device 3 for synchronizing the update service data to the search engine device 4 through a preset open source data synchronization tool Canal;
the search engine device 4 is configured to store the updated service data through a preset search and data analysis engine elastic search, perform separation processing and index creation processing on the updated service data, obtain service data to be queried, and query the service data to be queried according to a service data query request to obtain target query data.
Specifically, the application sending device 1 sends a service data update request to the data management device 2 according to a preset trigger instruction to call the data management device 2 to update the stored real-time service data to obtain updated service data, and sends a service data query request to the search engine device 4 to call the search engine device 4 to query the service data to be queried according to the service data query request to obtain target query data, so as to realize the separation of updating and querying of the service data, slow down the storage pressure and query pressure, and meet the requirements on the storage and running performance of a large amount of logistics service data and the requirements of multi-condition query, thereby enabling the system architecture of the real-time service system based on the elastic search to have higher flexibility.
The data management device 2 is used for storing real-time service data (within a single millions) and updating the real-time service data, wherein the service data is logistics service data, and comprises various operation data of a logistics order, such as: goods warehouse management data of logistics orders, trip data of transportation vehicles, goods dispatch status data and the like. Wherein, the data management device 2 may be built with one or more databases, and the multiple databases may be of the same type or different types, for example: a relational database MySQL is built on the data management device 2; or, the data management device 2 is built with a relational database MySQL and a relational database Oracle. By constructing a plurality of databases to carry out shunt parallel processing on the logistics business data, the data processing and data storage pressure of the data management device 2 is reduced, and the processing efficiency is improved, so that the performance of the real-time business system based on the elastic search is improved.
The database synchronization device 3 monitors the binary log of the service data in the relational database MySQL through the open source data synchronization tool Canal, and when the Canal monitors that the binary log of the service data has an update, synchronizes the service data having the update (i.e., update service data) to the search engine device 4. The database synchronization device 3 synchronizes the update service data of the relational database MySQL in the data management device 2 to the search engine device 4 through the open source data synchronization tool Canal, so as to reduce the storage and retrieval pressure of the real-time service system based on the elastic search and improve the processing efficiency and performance thereof.
The search engine device 4 performs separation processing and index creation processing on the updated service data through the updated service data in the data management device 2 synchronously written by the search and data analysis engine elastic search storage database synchronization device 3 to obtain service data to be queried, queries the service data to be queried according to the service data query request sent by the application sending device 1 to obtain target query data, and feeds the target query data back to the application sending device 1 to store historical service data formed by the fact that the real-time service data meets preset conditions, namely that the real-time service data is not in a preset logistics service data life cycle, so that read-write separation of a real-time service system based on the elastic search is realized, and the performance and flexibility of the system are improved, thereby improving the management and control performance of the real-time service system based on the elastic search on the real-time service data.
The search engine device 4 is further configured to create an index by using a preset search and data analysis engine elastic search to obtain the index, split the index into a plurality of sub-indexes, and name the plurality of sub-indexes by using aliases, where the index alias can be changed by calling a corresponding interface, so as to facilitate the query. The search engine device 4 is further configured to, after the index is created to obtain the index, segment the index according to a segmentation rule corresponding to the application type, for example: if the application type is log type (writing is more and reading is less), the index is sliced according to a slicing rule that the single slicing size is not more than 50G; if the application type is a search class (write-less read-more), the index is sliced with a slicing rule that the single slice size does not exceed 20G. The search engine device 4 is further configured to split the index according to the time dimension after the index is created to obtain the index, so as to facilitate management of update service data of an unused time period.
The search engine device 4 is further configured to, when receiving the service data query request sent by the application sending device 1 through the search and data analysis engine elastic search, analyze query information in the service data query request, where the query information may include a query time, a query item, and/or other query conditions. After the search engine device 4 analyzes the query information through the search and data analysis engine elastic search, the structure type of the query information is identified, and if the structure type is single, the query is directly carried out on the service data to be queried according to the query information in a single form; if the structure type is combined, carrying out multi-table query on the service data to be queried according to the preset query priority. The search engine device 4 is further configured to query, by using the search and data analysis engine elastic search, the service data to be queried according to the node status corresponding to the separated roles, for example: and the nodes corresponding to the separated roles are H1, H2 and H3, and if the H1 is in a downtime state, the H2 and the H3 are searched.
The application transmitting device 1 transmits a service data update request to the data management device 2 and a service data query request to the search engine device 4, so as to realize the storage and update operation of the real-time service data (updated service data in a preset service data life cycle) through the data management device 2, and the storage and query of the history service data (updated service data not in the preset service data life cycle) through the search engine device 4. The database synchronization device 3 is used for monitoring the change condition of the real-time service data in the data management device 2 in real time and synchronizing the real-time changed service data into the search engine device 4 in real time, so that near real-time synchronization between the data management device 2 and the search engine device 4 is ensured.
Referring to fig. 1 and 2, optionally, the database synchronization device 3 includes a monitor module 31, a buffer module 32, and a synchronization module 33; the monitoring module 31 is configured to analyze the incremental database log in the data management device 2 in real time, extract corresponding update service data, and send the update service data to the cache module 32; the buffer module 32 is configured to buffer the update service data, obtain buffered update service data, and send the buffered update service data to the synchronization module 33; the synchronization module 33 is configured to synchronize the cached update service data to the search engine device 4 according to the synchronization configuration file in the open source data synchronization tool Canal.
After the monitoring module 31 monitors the updated service data, the updated service data is sent to the caching module 32 for caching, and the cached updated service data is synchronized to the search engine device 4 through the synchronizing module 33 after caching. The synchronization profile may include, among other things, invoked interfaces, data format conversions, and synchronization rules, such as: for updating service data A, because the data volume is small, an interface with low relative performance is called to synchronously process the updating service data A. The monitoring module 31 is used for analyzing the incremental log of the database in the data management device 2 in real time and extracting the corresponding updated service data so as to synchronize the real-time changed logistics service data in the data management device 2 into the search engine device 4 in real time. Caching the updated service data by the caching module 32 so as to facilitate access to the updated service data, reduce access pressure to the database, and avoid message blocking; near real-time synchronization between the data management device 2 and the search engine device 4 is ensured by the synchronization module 33.
Referring to fig. 1 and 3, optionally, the search engine apparatus 4 includes a search and data analysis engine elastic search cluster including a main network node server 41 and a plurality of controlled network node servers 42; the master network node server 41 is respectively in communication connection with the plurality of controlled network node servers 42, and is used for controlling the plurality of controlled network node servers 42 to perform separation processing and index creation processing on the updated service data to obtain service data to be queried, and controlling the plurality of controlled network node servers 42 to query the service data to be queried according to the service data query request to obtain target query data.
The search engine device 4 includes a plurality of elastiscearch-based network node servers (search and data analysis engine elastiscearch clusters). The role separation and the load balancing of the search engine device 4 are realized by performing master-slave allocation cooperation on the plurality of controlled network node servers 42 through the master network node server 41 in the plurality of network node servers (search and data analysis engine) based on the elastic search, so that the storage of real-time service data and historical service data, the complex combined query and detail query are facilitated, the node resources are effectively utilized, the performance of the search engine device 4 is enhanced, and the processing efficiency of the search engine device 4 is improved.
Referring to fig. 1 and 3, the primary network node server 41 optionally includes a processing module 411;
a processing module 411, configured to perform hot node marking and cold node marking on the plurality of controlled network node servers 42 according to a preset service requirement, perform role node configuration on the plurality of controlled network node servers 42 according to preset role configuration information, and perform index creation on updated service data;
the processing module 411 includes a cold and hot separation sub-module 4111, an index creation sub-module 4112, a configuration sub-module 4113, a role separation sub-module 4114, and a transmission sub-module 4115;
The cold-hot separation submodule 4111 is configured to perform node marking on the plurality of controlled network node servers 42 according to a preset time node to obtain a hot node and a cold node, classify updated service data according to the hot node and the cold node to obtain hot node data and cold node data, and send the hot node data and the cold node data to the index creation submodule 4112, where the hot node corresponds to a preset target time period, the cold node corresponds to a period other than the target time period, and the target time period is a query time period determined according to a preset service requirement;
an index creation submodule 4112, configured to create an index for the hot node data and the cold node data, obtain node data with an created index, and send the node data with the created index to a sending submodule 4115;
a configuration submodule 4113, configured to store configuration information of each type of the search engine device 4, where the configuration information includes role configuration information, and send the role configuration information to the role separation submodule 4114;
a role separation submodule 4114, configured to perform role node configuration on the plurality of controlled network node servers 42 according to role configuration information, so as to obtain a plurality of configured controlled network node servers 42, where the role configuration information includes roles, nodes corresponding to the roles, and node configuration information of the nodes;
And the sending submodule 4115 is configured to send the updated service data to the corresponding controlled network node server 42 according to the hot node and the cold node, and send the node data with the created index to the configured controlled network node server 42 corresponding to the role according to the role and the node configuration information, so as to obtain the service data to be queried.
The preset service requirement is a query service requirement of a problem express, the problem express is an express with a situation in an express logistics, and the target time period is a query time period determined according to the preset service requirement, for example: the preset service requirement is the query service requirement of the problem express, the problem express is the express with conditions in the express logistics, the problem express has large data volume, and the main service data query request is concentrated in the last period R established by the order, and then R is the target period.
After the cold-hot separation sub-module 4111 performs node marking to obtain a hot node and a cold node, the sending sub-module 4115 sends the updated service data to the corresponding hot node or cold node, and the index creating sub-module 4112 performs index creation to realize cold-hot separation of the updated service data and meet the complex combined query requirement, so that the query on the service data to be queried in the search engine device 4 is convenient and efficient. The role separation sub-module 4114 is used for performing role node configuration on the plurality of controlled network node servers 42 according to the role configuration information sent by the configuration sub-module 4113, and the sending sub-module 4115 is used for sending the node data which is created by the index creation sub-module 4112 and is created to the corresponding controlled network node server 42 after the role separation sub-module 4114 performs the role node configuration, so that reasonable resource configuration is performed on the plurality of controlled network node servers 42, resource waste is reduced, load balancing is realized, system performance and processing efficiency are improved, and the problem that the management and control performance of the existing system framework on real-time service data is low is solved conveniently.
Referring to fig. 3 and 4, the index creation sub-module 4112 includes a first determination unit 41121, a first creation unit 41122, a transmission unit 41123, a second determination unit 41124, a migration unit 41125, and a third determination unit 41126;
a first determining unit 41121, configured to determine whether the updated service data is within a preset service data life cycle, determine the updated service data as history data if the updated service data is not within the preset service data life cycle, and send the history data to the sending unit 41123;
a first creating unit 41122, configured to create an index cluster on a hot node, obtain corresponding first node data, where the first node data includes a node index and a generation time of service data, and send the first node data to the second determining unit 41124 and the third determining unit 41126, respectively;
a sending unit 41123, configured to send the history data to the cold node, obtain target cold node data, and send the target cold node data to the migration unit 41125;
a second determining unit 41124, configured to obtain the current time, determine a node index to be migrated in the node indexes according to the generation time and the current time, and send the node index to be migrated to the migrating unit 41125;
The migration unit 41125 is configured to migrate, through a preset interface, the index of the node to be migrated to a cold node corresponding to the target cold node data, to obtain corresponding second node data;
the third determining unit 41126 is configured to determine the first node data obtained by the first creating unit 41122 and the second node data obtained by the migrating unit 41125 as node data having created an index.
The node index is used for inquiring the node corresponding to the hot node data and the service data corresponding to the node. The preset service data life cycle is the life cycle of the express logistics service data, for example: creation of express bill, warehouse management, logistics vehicle transportation, dispatch and after-sales customer service processing. The first determining unit 41121 determines whether the updated service data is within a preset service data life cycle, and if the updated service data is not within the preset service data life cycle, determines the updated service data as history data, for example: the last stage of the preset service data life cycle is an after-sales service processing stage, and the process node state of the updated service data is after the after-sales service processing stage (i.e. not within the preset service data life cycle), the first determining unit 41121 determines the updated service data as history data. The history data is updated service data not in a preset service data life cycle, and may be that a current flow node of the updated service data is a final flow node, and the current flow node is in an end state, for example: the final flow node is customer service arbitration management, the current flow node of the updated service data is customer service arbitration management, the running state is the ending state, and the customer service arbitration management is a stage outside the preset life cycle of the logistic service data, and the first determining unit 41121 determines the updated service data as history data.
If the updated service data is within the preset service data life cycle, the first determining unit 41121 determines the updated service data as real-time data, for example: one of the phases of the preset service data lifecycle is a dispatch management phase of the express, and the process node state of the updated service data is in the dispatch of the express (i.e. within the preset service data lifecycle), the first determining unit 41121 determines the updated service data as real-time data. The real-time data is updated service data in a preset service data life cycle, and may be that the current flow node of the updated service data is or is not the final flow node, and the current flow node is in operation, for example: the final flow node is customer service arbitration management, if the current flow node of the update service data is customer service arbitration management (before customer service arbitration management), and the running state is in progress, the first determining unit 41121 determines the update service data as real-time data, and if the current flow node of the update service data is customer service arbitration management, and the running state is in progress, the first determining unit 41121 determines the update service data as real-time data.
An index cluster is created on the hot node through the first creating unit 41122, the size of a single index is controlled to be within 20G or within 50G, so that the throughput and the delay of writing real-time service data into the index are balanced, corresponding first node data are sent to the second determining unit 41124, the current moment is obtained through the second determining unit 41124, the node index to be migrated in the node indexes is determined according to the generation moment and the current moment, the node index to be migrated is sent to the migrating unit 41125, and the node index to be migrated is migrated to the cold node through the migrating unit 41125. The history data is transmitted to the cold node by the transmitting unit 41123, resulting in target cold node data. The method and the device realize real-time storage and index management of the updated service data, and improve the query efficiency and accuracy, thereby facilitating and efficiently querying.
Referring to fig. 4 and 5, the second determining unit 41124 includes an acquiring subunit 411241, a calculating subunit 411242, a judging subunit 411243, and a determining subunit 411244;
an acquisition subunit 411241, configured to acquire the current time and send the current time to the calculation subunit 411242;
a calculating subunit 411242, configured to calculate a duration difference between the generation time and the current time, and send the duration difference to the judging subunit 411243;
The judging subunit 411243 is configured to judge whether the difference between the time lengths is greater than a preset threshold, obtain a judgment result, and send the judgment result to the determining subunit 411244;
the determining subunit 411244 is configured to determine, as the node index to be migrated, the node index corresponding to the first node data whose determination result is yes.
For example: the generation time of the service data is 2020.02.29, the current time acquired by the acquiring subunit 411241 is 2020.03.09, the difference between the time of generation and the current time is calculated by the calculating subunit 411242 to be 9 days, if the preset threshold is 5 days, the judging subunit 411243 sends the judging result that the difference between the time of generation and the current time is greater than the preset threshold to the determining subunit 411244, the determining subunit 411244 determines the node index corresponding to 2020.02.29 as the node index to be migrated, if the preset threshold is 10 days, the judging subunit 411243 sends the judging result that the difference between the time of generation and the current time is less than the preset threshold to the determining subunit 411244, and the determining subunit 411244 determines the node index corresponding to 2020.02.29 as the node index other than the node index to be migrated. The preset threshold may be set according to the target time period, that is, the preset threshold may be a duration corresponding to the target time period.
Referring to fig. 3 and 6, the index creation submodule 4112 includes a second creation unit 41127, a first setting unit 41128, an analysis unit 41129, a second setting unit 41120, and a fourth determination unit 411201;
a second creation unit 41127 for performing index creation on the hot node data and the cold node data to obtain an initial index, and transmitting the initial index to the first setting unit 41128;
a first setting unit 41128, configured to set the initial index in a sliced manner according to a preset number of main slices, obtain the sliced initial index, and send the sliced initial index to an analysis unit 41129;
an analysis unit 41129, configured to perform search requirement analysis, aggregation analysis and word segmentation requirement analysis on the segmented initial index to obtain candidate indexes, and send the candidate indexes to the second setting unit 41120;
a second setting unit 41120 configured to perform attribute setting and type setting on the candidate indexes, obtain target indexes, and send the target indexes to a fourth determining unit 411201;
the fourth determining unit 411201 is configured to determine hot node data created with the target index and cold node data created with the target index as node data having the created index.
After the second creation unit 41127 performs index creation on the hot node data and the cold node data, the obtained initial index is sent to the first setting unit 41128, the initial index is set in pieces by the first setting unit 41128, the obtained initial index trigger after being split is sent to the analysis unit 41129, analyzed by the analysis unit 41129, and the obtained candidate index trigger is sent to the second setting unit 41120, and index attribute setting and index type setting are performed by the second setting unit 41120, thereby obtaining the target index.
The first setting unit 41128 is used for performing slicing setting on the initial index according to the preset main slicing number, so that the maximum number of slices of the initial index on one node can be set, and the situation that the split index slices of the initial index are intensively distributed to a small number of nodes to cause unbalanced load is avoided. The search requirement analysis, the aggregation analysis and the word segmentation requirement analysis are performed on the fields in the segmented initial index through the analysis unit 41129 to obtain candidate indexes, and the attribute setting and the type setting are performed on the candidate indexes through the second setting unit 41120, so that the data writing performance is improved, and the efficiency of creating and searching the indexes is improved.
Referring to fig. 1 and 3, the primary network node server 41 further includes an optional query module 412;
a query module 412, configured to query the service data to be queried in the plurality of controlled network node servers 42 according to the service data query request, to obtain target query data;
the query module 412 includes an extraction submodule 4121, an analysis submodule 4122, and a retrieval submodule 4123;
an extracting submodule 4121, configured to receive the service data query request sent by the application sending device 1, extract query time information in the service data query request, and send the query time information to an analyzing submodule 4122;
an analysis submodule 4122, configured to perform node analysis on the configured plurality of controlled network node servers 42 obtained by the processing module 411 according to the query time information and the role configuration information, so as to obtain corresponding target nodes;
the retrieval submodule 4123 is configured to retrieve node data of the created index in the target node obtained by the analysis submodule 4122 according to the service data query request, so as to obtain target query data corresponding to the service data query request.
The extraction submodule 4121 is used for extracting the query time information in the service data query request sent by the application sending device 1, the query time information is sent to the analysis submodule 4122, the analysis submodule 4122 is used for carrying out node analysis, so that the node which is firstly searched (namely the target node) is determined, and then the searching submodule 4123 is used for correspondingly searching at the target node according to the query request, so that the load balancing of the search engine device 4 is realized, the low query efficiency caused by the load operation on the network node server with high resource occupation ratio is avoided, and the reasonable utilization of resources and the improvement of the query efficiency are facilitated.
Referring to fig. 3 and 7, optionally, the analysis submodule 4122 includes a sequence control unit 41221, a time analysis unit 41222, and a configuration information analysis unit 41223; a sequence control unit 41221 for controlling the analysis sequence of the time analysis unit 41222 and the configuration information analysis unit 41223. The time analysis unit 41222 is configured to determine whether to retrieve the node data of the hot node or the node data of the cold node according to the query time information, so as to obtain an initial node. The configuration information analysis unit 41223 is configured to obtain the resource ratio of each network node server in the search engine device 4, and determine the target node in the initial node according to the resource ratio and the role configuration information of each network node server. So as to conveniently, efficiently and orderly perform inquiry.
Referring to fig. 8, the optional real-time service system based on the elastic search further includes an optimization module 5, where the optimization module 5 is configured to record and statistically analyze the service data query request and the target query data to obtain statistical analysis data, and optimize the cold-hot separation of the updated service data in the search engine device 4 according to the statistical analysis data.
The optimization module 5 is communicatively connected to the search engine device 4. The optimization module 5 comprises a logging sub-module 51, a statistical analysis sub-module 52 and an optimization sub-module 53. The recording sub-module 51 is configured to record each service data query request and each target query data, obtain record data, and send the record data to the statistical analysis sub-module 52. The statistical analysis sub-module 52 is configured to perform statistical analysis on the record data to obtain statistical analysis data, where the statistical analysis data may be a query time period with a greater number of queries, and send the statistical analysis data to the optimization sub-module 53. The optimizing sub-module 53 is configured to adjust a time period of a preset time node of the cold-hot separation according to the statistical analysis data, so as to optimize the cold-hot separation of the updated service data in the search engine device 4, so as to improve the accuracy of the cold-hot data separation of the updated service data, thereby improving the query accuracy of the service data to be queried.
Optionally, the search engine device 4 further includes a parameter setting module 43, where the parameter setting module 43 is configured to set parameters in the search engine device 4, where the parameters include a refresh time interval, index shard data, the number of shards on a single node, a storage location and a storage manner of a transaction log, and an index setting policy.
The setting of the refresh time interval, the index fragment data, the number of fragments on a single node, the storage location and storage mode of the transaction log, and the setting policy of the index are performed by the parameter setting module 43, so that the search engine device 4 efficiently configures the scheme of updating the service data processing. The performance and processing efficiency of the search engine device 4 are improved and load balancing is achieved.
In the embodiment of the invention, the real-time business system based on the elastic search is provided, the requirements on the storage and running performance of a large amount of logistics business data, the requirement on multi-condition query and the requirement on the life cycle management of the business data are met through a data management device, a database synchronization device and a search engine device, the synchronization, the reading and writing separation of the logistics business data are realized, and the real-time data is processed through the search engine device based on the elastic search, so that the storage and query modes of the real-time data can be selected in a single-table mode or a multi-table mode, the system architecture has higher flexibility, and the problem that the management and control performance of the conventional system framework on the real-time business data is low is solved.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. The real-time business system based on the elastic search is characterized by comprising an application sending device, a data management device, a database synchronization device and a search engine device; wherein:
the application sending device is used for sending a service data updating request to the data management device and sending a service data query request to the search engine device;
the data management device is used for carrying out updating processing on the stored real-time service data according to the service data updating request to obtain updated service data, and sending the updated service data to the database synchronization device, wherein the service data is logistics service data;
The database synchronization device is used for synchronizing the update service data to the search engine device through a preset open source data synchronization tool Canal;
the search engine device is used for storing the updated service data through a preset search and data analysis engine elastic search, separating and creating the updated service data to obtain service data to be queried, and querying the service data to be queried according to the service data query request to obtain target query data;
the search engine device comprises a search and data analysis engine elastsearch cluster, wherein the search and data analysis engine elastsearch cluster comprises a main network node server and a plurality of controlled network node servers;
the master network node server is respectively in communication connection with the plurality of controlled network node servers and is used for controlling the plurality of controlled network node servers to perform separation processing and index creation processing on the updated service data to obtain service data to be queried, and controlling the plurality of controlled network node servers to query the service data to be queried according to the service data query request to obtain target query data;
The master network node server comprises a processing module;
the processing module is used for marking hot nodes and cold nodes of the controlled network node servers according to preset service requirements, performing role node configuration on the controlled network node servers according to preset role configuration information, and performing index creation on the updated service data;
the processing module comprises a cold and hot separation sub-module, an index creation sub-module, a configuration sub-module, a role separation sub-module and a sending sub-module;
the cold-hot separation sub-module is configured to perform node marking on the plurality of controlled network node servers according to a preset time node to obtain a hot node and a cold node, classify the updated service data according to the hot node and the cold node to obtain hot node data and cold node data, and send the hot node data and the cold node data to the index creation sub-module, where the hot node corresponds to a preset target time period, the cold node corresponds to a time period outside the target time period, and the target time period is a query time period determined according to a preset service requirement;
The index creation sub-module is used for carrying out index creation on the hot node data and the cold node data to obtain node data with created indexes, and sending the node data with created indexes to the sending sub-module;
the configuration sub-module is used for storing configuration information of each type of the search engine device, wherein the configuration information comprises role configuration information and sends the role configuration information to the role separation sub-module;
the role separation module is configured to perform role node configuration on the plurality of controlled network node servers according to the role configuration information to obtain a plurality of configured controlled network node servers, wherein the role configuration information comprises a role, a node corresponding to the role and node configuration information of the node;
the sending submodule is used for sending the updated service data to the corresponding controlled network node server according to the hot node and the cold node, and sending the node data with the created index to the configured controlled network node server corresponding to the role according to the role and the node configuration information, so as to obtain the service data to be queried.
2. The elastic search based real-time service system according to claim 1, wherein the index creation sub-module comprises a first determination unit, a first creation unit, a transmission unit, a second determination unit, a migration unit, and a third determination unit;
the first determining unit is configured to determine whether the updated service data is within a preset service data life cycle, determine the updated service data as historical data if the updated service data is not within the preset service data life cycle, and send the historical data to the sending unit;
the first creating unit is configured to create an index cluster on the hot node, obtain corresponding first node data, and send the first node data to the second determining unit and the third determining unit, where the first node data includes a node index and a generation time of service data;
the sending unit is used for sending the historical data to the cold node to obtain target cold node data, and sending the target cold node data to the migration unit;
the second determining unit is configured to obtain a current time, determine a node index to be migrated in the node indexes according to the generation time and the current time, and send the node index to be migrated to the migration unit;
The migration unit is used for migrating the node index to be migrated to the cold node corresponding to the target cold node data through a preset interface to obtain corresponding second node data;
the third determining unit is configured to determine the first node data obtained by the first creating unit and the second node data obtained by the migration unit as node data having created an index.
3. The elastic search based real-time service system according to claim 1, wherein the index creation sub-module comprises a second creation unit, a first setting unit, an analysis unit, a second setting unit, and a fourth determination unit;
the second creating unit is configured to create an index for the hot node data and the cold node data, obtain an initial index, and send the initial index to the first setting unit;
the first setting unit is used for performing slicing setting on the initial index according to a preset main slicing number to obtain an initial index after slicing, and sending the initial index after slicing to the analysis unit;
the analysis unit is used for carrying out search requirement analysis, aggregation analysis and word segmentation requirement analysis on the segmented initial indexes to obtain candidate indexes, and sending the candidate indexes to the second setting unit;
The second setting unit is configured to perform attribute setting and type setting on the candidate indexes to obtain target indexes, and send the target indexes to the fourth determining unit;
the fourth determining unit is configured to determine hot node data created with the target index and cold node data created with the target index as node data of the created index.
4. The elastic search based real time business system of claim 1, wherein said primary network node server further comprises a query module;
the query module is used for querying the service data to be queried in the plurality of controlled network node servers according to the service data query request to obtain target query data;
the query module comprises an extraction submodule, an analysis submodule and a retrieval submodule;
the extraction sub-module is used for receiving the service data query request sent by the application sending device, extracting the query time information in the service data query request, and sending the query time information to the analysis sub-module;
the analysis submodule is used for carrying out node analysis on the plurality of configured controlled network node servers obtained by the processing module according to the inquiry time information and the role configuration information to obtain corresponding target nodes;
And the retrieval submodule is used for retrieving node data with created indexes in the target nodes obtained by the analysis submodule according to the service data query request to obtain target query data corresponding to the service data query request.
5. The elastic search based real-time service system according to claim 2, wherein the second determining unit comprises an acquiring subunit, a calculating subunit, a judging subunit, and a determining subunit;
the obtaining subunit is used for obtaining the current time and sending the current time to the calculating subunit;
the calculating subunit is used for calculating a time length difference value between the generating time and the current time and sending the time length difference value to the judging subunit;
the judging subunit is configured to judge whether the duration difference is greater than a preset threshold, obtain a judging result, and send the judging result to the determining subunit;
and the determining subunit is configured to determine, as the node index to be migrated, the node index corresponding to the first node data whose determination result is yes.
6. The real-time business system based on elastic search according to claim 1, wherein the database synchronization device comprises a monitoring module, a buffering module and a synchronization module;
The monitoring module is used for analyzing the incremental log of the database in the data management device in real time, extracting corresponding updating service data and sending the updating service data to the cache module;
the buffer module is used for buffering the update service data to obtain buffered update service data and sending the buffered update service data to the synchronization module;
and the synchronization module is used for synchronizing the cached update service data to the search engine device according to the synchronization configuration file in the open source data synchronization tool Canal.
7. The real-time business system based on the elastic search according to claim 1, further comprising an optimizing module, wherein the optimizing module is used for recording and statistically analyzing the business data query request and the target query data to obtain statistical analysis data, and optimizing cold-hot separation of updated business data in the search engine device according to the statistical analysis data.
8. The elastic search based real time business system as in any one of claims 1 to 7, wherein said search engine means further comprises a parameter setting module for setting parameters in said search engine means, said parameters comprising refresh time interval, index shard data, number of shards on a single node, storage location and storage means of transaction log and setting policy of index.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010358051.7A CN111460024B (en) | 2020-04-29 | 2020-04-29 | Real-time service system based on elastic search |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010358051.7A CN111460024B (en) | 2020-04-29 | 2020-04-29 | Real-time service system based on elastic search |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111460024A CN111460024A (en) | 2020-07-28 |
| CN111460024B true CN111460024B (en) | 2023-06-09 |
Family
ID=71680672
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010358051.7A Active CN111460024B (en) | 2020-04-29 | 2020-04-29 | Real-time service system based on elastic search |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111460024B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112435022B (en) * | 2020-11-19 | 2023-09-22 | 上海领健信息技术有限公司 | Dynamic retrieval system and method based on user real-time data |
| CN112749197B (en) * | 2021-01-12 | 2024-04-05 | 中国平安财产保险股份有限公司 | Data fragment refreshing method, device, equipment and storage medium |
| CN112883252A (en) * | 2021-02-05 | 2021-06-01 | 成都新希望金融信息有限公司 | Service query method, device, computer equipment and readable storage medium |
| CN113220756A (en) * | 2021-03-25 | 2021-08-06 | 上海东普信息科技有限公司 | Logistics data real-time processing method, device, equipment and storage medium |
| CN113570433A (en) * | 2021-07-28 | 2021-10-29 | 上海帘邦信息科技有限公司 | Integrated management system applied to non-calibration product industry |
| CN113468199B (en) * | 2021-07-29 | 2022-11-04 | 上海哔哩哔哩科技有限公司 | Index updating method and system |
| CN113641761B (en) * | 2021-08-20 | 2024-07-12 | 北京达佳互联信息技术有限公司 | Data synchronization method and device |
| CN115840788B (en) * | 2023-02-21 | 2023-04-28 | 创意信息技术股份有限公司 | Method, device, terminal and storage medium for synchronizing MySql data to ES |
| CN116955417A (en) * | 2023-09-19 | 2023-10-27 | 武汉大数据产业发展有限公司 | Optimization method and device for multi-table combined retrieval of data and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016082608A1 (en) * | 2014-11-25 | 2016-06-02 | 中兴通讯股份有限公司 | Service data query/search method, apparatus and system |
| CN110196851A (en) * | 2019-05-09 | 2019-09-03 | 腾讯科技（深圳）有限公司 | A kind of date storage method, device, equipment and storage medium |
| CN110569302A (en) * | 2019-08-16 | 2019-12-13 | 苏宁云计算有限公司 | method and device for physical isolation of distributed cluster based on lucene |
-
2020
- 2020-04-29 CN CN202010358051.7A patent/CN111460024B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016082608A1 (en) * | 2014-11-25 | 2016-06-02 | 中兴通讯股份有限公司 | Service data query/search method, apparatus and system |
| CN110196851A (en) * | 2019-05-09 | 2019-09-03 | 腾讯科技（深圳）有限公司 | A kind of date storage method, device, equipment and storage medium |
| CN110569302A (en) * | 2019-08-16 | 2019-12-13 | 苏宁云计算有限公司 | method and device for physical isolation of distributed cluster based on lucene |
Non-Patent Citations (1)
| Title |
|---|
| 姜康 ; 冯钧 ; 唐志贤 ; 王超 ; .基于ElasticSearch的元数据搜索与共享平台.计算机与现代化.2015,(02),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111460024A (en) | 2020-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111460024B (en) | Real-time service system based on elastic search | |
| CN111460023B (en) | Method, device, equipment and storage medium for processing service data based on elastic search | |
| CN102521405B (en) | Massive structured data storage and query methods and systems supporting high-speed loading | |
| CN102521406B (en) | Distributed query method and system for complex task of querying massive structured data | |
| US9609060B2 (en) | Distributed storage system and method | |
| CN109063196B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| US10223437B2 (en) | Adaptive data repartitioning and adaptive data replication | |
| CN112307037A (en) | Data synchronization method and device | |
| CN107038162A (en) | Real time data querying method and system based on database journal | |
| CN110245134B (en) | Increment synchronization method applied to search service | |
| CN103970902A (en) | Method and system for reliable and instant retrieval on situation of large quantities of data | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN104978324A (en) | Data processing method and device | |
| CN102054000A (en) | Data querying method, device and system | |
| CN104317957A (en) | Open platform and system for processing reports and report processing method | |
| CN111858760A (en) | Data processing method and device for heterogeneous database | |
| KR101955376B1 (en) | Processing method for a relational query in distributed stream processing engine based on shared-nothing architecture, recording medium and device for performing the method | |
| CN103034650A (en) | System and method for processing data | |
| CN113672583B (en) | Big data multi-data source analysis method and system based on storage and calculation separation | |
| CN106776810B (en) | Big data processing system and method | |
| CN117633116A (en) | Data synchronization method, device, electronic equipment and storage medium | |
| CN117076426A (en) | Traffic intelligent engine system construction method and device based on flow batch integration | |
| CN114063931B (en) | Data storage method based on big data | |
| CN110347726A (en) | A kind of efficient time series data is integrated to store inquiry system and method | |
| CN113590651B (en) | HQL-based cross-cluster data processing system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |