CN111460023A - Service data processing method, device, equipment and storage medium based on elastic search - Google Patents
Service data processing method, device, equipment and storage medium based on elastic search Download PDFInfo
- Publication number
- CN111460023A CN111460023A CN202010358044.7A CN202010358044A CN111460023A CN 111460023 A CN111460023 A CN 111460023A CN 202010358044 A CN202010358044 A CN 202010358044A CN 111460023 A CN111460023 A CN 111460023A
- Authority
- CN
- China
- Prior art keywords
- data
- service data
- node
- index
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24532—Query optimisation of parallel queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/083—Shipping
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Business, Economics & Management (AREA)
- Economics (AREA)
- Human Resources & Organizations (AREA)
- Entrepreneurship & Innovation (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Quality & Reliability (AREA)
- Operations Research (AREA)
- Marketing (AREA)
- Tourism & Hospitality (AREA)
- Development Economics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the technical field of logistics, and provides a service data processing method, a device, equipment and a storage medium based on an elastic search, which are used for solving the problem that the conventional logistics service data processing scheme cannot meet the service object processing logic requirements of a large amount of logistics service data. The service data processing method based on the Elasticissearch comprises the following steps: receiving a service data updating request, and updating service data in a preset relational database according to the service data updating request to obtain updated service data, wherein the service data is logistics service data; writing the updated service data into a preset search engine Elasticissearch cluster through a preset open source data synchronization tool (cancer); performing cold-hot separation, index creation and role separation on the updated service data in sequence to obtain service data to be inquired; and receiving a service data query request, and querying service data to be queried in the Elasticissearch cluster of the search engine according to the service data query request to obtain target query data.
Description
Technical Field
The invention relates to the technical field of logistics, in particular to a service data processing method, a device, equipment and a storage medium based on an elastic search.
Background
With the rapid development of internet technology and in the environment of big data, in order to manage and control business data, the logistics industry uses a plurality of management type logistics business systems in respective systems, and the business systems have a large number of logistics business object processing logical phenomena, such as: and performing operations of inserting, modifying, deleting and inquiring the logistics business data. The prior logistics business data processing scheme deals with a large amount of logistics business object processing logic phenomena by storing business data in a structured database.
However, as the logistics traffic increases, the logistics business data volume in the structured database also increases, and when the data scale of the logistics business data volume reaches hundreds of millions, billions or billions, the structured database cannot meet the storage and operation performance requirements of big data and the requirements of multi-condition query, so that the existing logistics business data processing scheme cannot meet the business object processing logic requirements of a large amount of logistics business data.
Disclosure of Invention
The invention mainly aims to solve the problem that the conventional logistics business data processing scheme cannot meet the business object processing logic requirements of a large amount of logistics business data.
The invention provides a service data processing method based on an elastic search in a first aspect, which comprises the following steps:
receiving a service data updating request, and updating service data in a preset relational database according to the service data updating request to obtain updated service data, wherein the service data is logistics service data;
writing the updated service data into a preset search engine Elasticissearch cluster through a preset open source data synchronization tool (cancer);
performing cold-hot separation, index creation and role separation on the updated service data in sequence to obtain service data to be inquired;
and receiving a service data query request, and querying service data to be queried in the search engine Elasticissearch cluster according to the service data query request to obtain target query data.
Optionally, in a first implementation manner of the first aspect of the present invention, the performing cold-hot separation, index creation, and role separation on the updated service data in sequence to obtain service data to be queried includes:
carrying out node marking on the updated service data according to a preset time node to obtain hot node data and cold node data, wherein the hot node data corresponds to a preset target time period, the cold node data corresponds to a time period other than the target time period, and the target time period is a query time period determined according to a preset service requirement;
index creation is carried out on the hot node data and the cold node data, and node data with created indexes are obtained;
acquiring role configuration information corresponding to the search engine Elasticissearch cluster, wherein the role configuration information comprises roles, nodes corresponding to the roles and node configuration information of the nodes;
and sending the node data with the created index to a node corresponding to the role according to the role and the node configuration information to obtain service data to be inquired.
Optionally, in a second implementation manner of the first aspect of the present invention, the creating an index for the hot node data and the cold node data to obtain indexed node data includes:
judging whether the updated service data is in a preset service data life cycle;
if the updated service data is not in the preset service data life cycle, determining the updated service data as historical data;
creating an index cluster on a node corresponding to the hot node data to obtain corresponding first node data, wherein the first node data comprises a node index and the generation time of service data;
sending the historical data to a node corresponding to the cold node data to obtain target cold node data;
acquiring the current moment, and determining a node index to be migrated in the node indexes according to the generation moment and the current moment;
migrating the node index to be migrated to a node corresponding to the target cold node data through a preset interface to obtain corresponding second node data;
determining the first node data and the second node data as indexed node data.
Optionally, in a third implementation manner of the first aspect of the present invention, the creating an index for the hot node data and the cold node data to obtain indexed node data includes:
index creation is carried out on the hot node data and the cold node data to obtain an initial index;
carrying out fragmentation setting on the initial index according to a preset number of main fragments to obtain a fragmented initial index;
performing retrieval demand analysis, aggregation analysis and word segmentation demand analysis on the fragmented initial index to obtain a candidate index;
performing attribute setting and type setting on the candidate index to obtain a target index;
and determining the hot node data created with the target index and the cold node data created with the target index as the node data of the created index.
Optionally, in a fourth implementation manner of the first aspect of the present invention, the receiving a service data query request, and querying service data to be queried in the search engine Elasticsearch cluster according to the service data query request to obtain target query data includes:
receiving a service data query request, and extracting query time information in the service data query request;
analyzing nodes corresponding to the hot node data and the cold node data according to the query time information and the role configuration information to obtain corresponding target nodes;
and retrieving the node data of which the index is created in the target node to obtain target query data corresponding to the service data query request.
Optionally, in a fifth implementation manner of the first aspect of the present invention, the writing, by a preset open source data synchronization tool Canal, the updated service data into a preset search engine elastic search cluster includes:
extracting the updated service data through a preset open source data synchronization tool (cancer), and sending the updated service data to a preset cache region to obtain cached updated service data;
and synchronizing the cached updated service data to a preset search engine Elasticissearch cluster according to a synchronization configuration file in the open source data synchronization tool Canal.
Optionally, in a sixth implementation manner of the first aspect of the present invention, after receiving a service data query request, and querying service data to be queried in the search engine elastic search cluster according to the service data query request to obtain target query data, the method further includes:
and recording and statistically analyzing the business data query request and the target query data to obtain statistical analysis data, wherein the statistical analysis data is used for optimizing cold-hot separation of the updated business data.
The second aspect of the present invention provides an elastic search based service data processing apparatus, including:
the updating module is used for receiving a service data updating request and updating service data in a preset relational database according to the service data updating request to obtain updated service data, wherein the service data is logistics service data;
the writing module is used for writing the updated service data into a preset search engine Elasticissearch cluster through a preset open source data synchronization tool (cancer);
the processing module is used for sequentially carrying out cold-hot separation, index creation and role separation on the updated service data to obtain service data to be inquired;
and the query module is used for receiving a service data query request and querying the service data to be queried in the search engine Elasticissearch cluster according to the service data query request to obtain target query data.
Optionally, in a first implementation manner of the second aspect of the present invention, the processing module includes:
the node marking unit is used for marking the nodes of the updated service data according to preset time points to obtain hot node data and cold node data, wherein the hot node data corresponds to a preset target time period, the cold node data corresponds to a time period except the target time period, and the target time period is a query time period determined according to preset service requirements;
the index creating unit is used for creating an index for the hot node data and the cold node data to obtain node data with the created index;
the acquiring unit is used for acquiring role configuration information corresponding to the search engine Elasticissearch cluster, wherein the role configuration information comprises roles, nodes corresponding to the roles and node configuration information of the nodes;
and the sending unit is used for sending the node data with the created index to the node corresponding to the role according to the role and the node configuration information to obtain the service data to be inquired.
Optionally, in a second implementation manner of the second aspect of the present invention, the index creating unit is specifically configured to:
judging whether the updated service data is in a preset service data life cycle;
if the updated service data is not in the preset service data life cycle, determining the updated service data as historical data;
creating an index cluster on a node corresponding to the hot node data to obtain corresponding first node data, wherein the first node data comprises a node index and the generation time of service data;
sending the historical data to a node corresponding to the cold node data to obtain target cold node data;
acquiring the current moment, and determining a node index to be migrated in the node indexes according to the generation moment and the current moment;
migrating the node index to be migrated to a node corresponding to the target cold node data through a preset interface to obtain corresponding second node data;
determining the first node data and the second node data as indexed node data.
Optionally, in a third implementation manner of the second aspect of the present invention, the index creating unit may be further specifically configured to:
index creation is carried out on the hot node data and the cold node data to obtain an initial index;
carrying out fragmentation setting on the initial index according to a preset number of main fragments to obtain a fragmented initial index;
performing retrieval demand analysis, aggregation analysis and word segmentation demand analysis on the fragmented initial index to obtain a candidate index;
performing attribute setting and type setting on the candidate index to obtain a target index;
and determining the hot node data created with the target index and the cold node data created with the target index as the node data of the created index.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the query module is specifically configured to:
receiving a service data query request, and extracting query time information in the service data query request;
analyzing nodes corresponding to the hot node data and the cold node data according to the query time information and the role configuration information to obtain corresponding target nodes;
and retrieving the node data of which the index is created in the target node to obtain target query data corresponding to the service data query request.
Optionally, in a fifth implementation manner of the second aspect of the present invention, the writing module is specifically configured to:
extracting the updated service data through a preset open source data synchronization tool (cancer), and sending the updated service data to a preset cache region to obtain cached updated service data;
and synchronizing the cached updated service data to a preset search engine Elasticissearch cluster according to a synchronization configuration file in the open source data synchronization tool Canal.
Optionally, in a sixth implementation manner of the second aspect of the present invention, the service data processing apparatus based on an Elasticsearch further includes:
and the record counting module is used for recording and carrying out statistical analysis on the service data query request and the target query data to obtain statistical analysis data, and the statistical analysis data is used for optimizing the cold-hot separation of the updated service data.
A third aspect of the present invention provides an elastic search based service data processing device, including: a memory having instructions stored therein and at least one processor, the memory and the at least one processor interconnected by a line; the at least one processor calls the instruction in the memory to enable the Elasticissearch-based business data processing device to execute the Elasticissearch-based business data processing method.
A fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions that, when executed on a computer, cause the computer to execute the above-mentioned method for processing service data based on Elasticsearch.
In the technical scheme provided by the invention, a service data updating request is received, and service data in a preset relational database is updated according to the service data updating request to obtain updated service data, wherein the service data is logistics service data; writing the updated service data into a preset search engine Elasticissearch cluster through a preset open source data synchronization tool (cancer); performing cold-hot separation, index creation and role separation on the updated service data in sequence to obtain service data to be inquired; and receiving a service data query request, and querying service data to be queried in the search engine Elasticissearch cluster according to the service data query request to obtain target query data. In the invention, the logistics service data in the preset relational database is updated, the updated logistics service data is synchronized to the search engine Elasticissearch cluster through the open source data synchronization tool, and the search engine Elasticissearch cluster is queried, so that the requirements on storage and operation performance of a large amount of logistics service data and the requirements on multi-condition query are met, the synchronization and read-write separation of the logistics service data are realized, and the problem that the conventional logistics service data processing scheme cannot meet the service object processing logic requirements of the large amount of logistics service data is solved.
Drawings
Fig. 1 is a schematic diagram of an embodiment of an elastic search based service data processing method in an embodiment of the present invention;
fig. 2 is a schematic diagram of another embodiment of an elastic search based service data processing method in the embodiment of the present invention;
fig. 3 is a schematic diagram of an embodiment of an elastic search based service data processing apparatus according to an embodiment of the present invention;
fig. 4 is a schematic diagram of another embodiment of an elastic search based service data processing apparatus according to an embodiment of the present invention;
fig. 5 is a schematic diagram of an embodiment of an elastic search based service data processing device in the embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a service data processing method, a device, equipment and a storage medium based on an elastic search, and solves the problem that the conventional logistics service data processing scheme cannot meet the service object processing logic requirements of a large amount of service data.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of understanding, a specific flow of the embodiment of the present invention is described below, and referring to fig. 1, an embodiment of the method for processing service data based on an Elasticsearch in the embodiment of the present invention includes:
101. and receiving a service data updating request, and updating service data in a preset relational database according to the service data updating request to obtain updated service data, wherein the service data is logistics service data.
It can be understood that the execution subject of the present invention may be an Elasticsearch-based service data processing apparatus, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
When a server receives a business data updating request sent by a user side, analyzing a business data updating type contained in the business data updating request, wherein the business data updating type can be at least one of inserting, modifying, deleting and replacing the business data, the business data is logistics business data and comprises various operation data of a logistics order, such as goods warehouse management data of the logistics order, travel data of a transport vehicle, goods delivery condition data and the like, wherein a preset relational database is MySQ L, real-time data (within million level of a single table) is stored through the preset relational database MySQ L, and the inserting, modifying or other updating operations of the real-time data are met.
It should be noted that, before the server receives the service data update request and updates the service data in the preset relational database MySQ L according to the service data update request, the server performs parameter setting, where the parameters may include, but are not limited to, a Refresh Interval, an index buffer size, and a write position of the transaction log, for example, setting a Refresh Interval to adjust the frequency of Refresh trigger, if the Refresh Interval is set to-1, disabling automatic Refresh, setting the size of the index buffer by using a parameter index.
102. And writing the updated service data into a preset search engine Elasticissearch cluster through a preset open source data synchronization tool (cancer).
The server calls a preset open source data synchronization tool, monitors a binary log of the business data in the preset relational database MySQ L through the open source data synchronization tool, and synchronizes the business data (namely updated business data) with update to a preset search engine Elastic search cluster when the binary log of the business data is monitored to be updated by the cancer.
103. And performing cold-hot separation, index creation and role separation on the updated service data in sequence to obtain the service data to be inquired.
The server sets a time node in advance according to a service requirement, divides updated service data into hot data and cold data according to the time node, and performs node configuration according to the hot data and the cold data, for example: and processing hot data by using the nodes with relatively high configuration, and processing cold data by using the nodes with relatively low configuration so as to reasonably distribute node resources and improve the processing efficiency. The business requirement can be a question query requirement in the logistics order or a statistical requirement of the logistics order in a specific time period.
After the server performs cold-hot separation on the updated service data, index creation is performed on the hot data and the cold data after the cold-hot separation, an index of a multi-combination type query structure is created, and after the index is obtained, the server performs fragmentation on the index according to a fragmentation rule corresponding to an application type, for example: if the application type is the log type (writing more and reading less), the index is fragmented by a fragmentation rule that the size of a single fragmentation does not exceed 50G; and if the application type is the search type (writing less, reading more), the index is fragmented by a fragmentation rule that the size of a single fragmentation does not exceed 20G.
It should be noted that, after the server obtains the index, the index is split according to the time dimension, so as to facilitate management of the updated service data in the unused time period. After the server obtains the index, the index is divided into a plurality of sub-indexes, and the sub-indexes are named by using aliases, so that the query is facilitated, and the alias of the index is changed by calling a corresponding interface. And performing role separation processing on roles in the updated service data according to the node configuration information, thereby obtaining the service data to be inquired. Through role separation, node resources are effectively utilized, performance is enhanced, and processing efficiency is improved.
104. And receiving a service data query request, and querying service data to be queried in the Elasticissearch cluster of the search engine according to the service data query request to obtain target query data.
When receiving a service data query request sent by a user, a server analyzes query information in the service data query request, where the query information may include query time, query terms, and/or other query conditions. And after analyzing the query information, the server identifies the structure type of the query information, and directly performs single-table query on the service data to be queried in the Elasticissearch cluster of the search engine according to the query information if the structure type is single. If the structure type is combined, performing multi-table query on the service data to be queried in the Elasticsearch cluster of the search engine according to a preset query priority, for example: the structure type of the query information is combined, the query information comprises query time D, a delivery place E and a delivery place F, and the query priority is as follows: time-first priority, address-second priority and order state-finally, inquiring service data to be inquired in the Elasticissearch cluster of the search engine according to D to obtain inquiry data G1, inquiring G1 according to E to obtain G2, and inquiring G2 according to F to obtain G3 (target inquiry data).
It should be noted that the server may also query the service data to be queried in the Elasticsearch cluster of the search engine according to the node status after the role separation, for example: and after the roles are separated, the corresponding nodes are H1, H2 and H3, and if the H1 is in a down state, the H2 and the H3 are searched.
In the embodiment of the invention, the logistics service data in the preset relational database are updated, the updated logistics service data are synchronized to the search engine Elasticissearch cluster through the open source data synchronization tool Canal, and the search engine Elasticissearch cluster is queried, so that the requirements on storage and operation performance of a large amount of logistics service data and the requirements on multi-condition query are met, the logistics service data are synchronized and read-write separated, and the problem that the conventional logistics service data processing scheme cannot meet the service object processing logic requirements of a large amount of logistics service data is solved.
Referring to fig. 2, another embodiment of the service data processing method based on the elastic search in the embodiment of the present invention includes:
201. and receiving a service data updating request, and updating service data in a preset relational database according to the service data updating request to obtain updated service data, wherein the service data is logistics service data.
After receiving the update operation instruction of the service data update request, the preset relational database MySQ L may perform corresponding replacement update on the stored logistics service data according to the new service data in the update operation instruction, or perform operations such as insertion, modification, deletion and the like on the stored logistics service data according to the update operation instruction, thereby obtaining the updated service data.
It should be noted that, when receiving the logistics service data to be processed, the preset relational database MySQ L backs up the logistics service data to be processed to the slave library, and when detecting that the update service data has been processed (synchronized), the data in the slave library is cleared to prevent data loss and data missing.
202. And writing the updated service data into a preset search engine Elasticissearch cluster through a preset open source data synchronization tool (cancer).
Specifically, the server extracts the updated service data through a preset open source data synchronization tool, and sends the updated service data to a preset cache region to obtain cached updated service data; and according to a synchronization configuration file in an open source data synchronization tool, synchronizing the cached updated service data to a preset search engine Elasticissearch cluster.
The server monitors the binary log of the service data in the preset relational database MySQ L through an open source data synchronization tool, and when the Canal monitors that the binary log of the service data is updated, extracts the updated service data (namely, the updated service data) and sends the updated service data to a preset cache region to cache the updated service data so as to obtain the cached updated service data.
203. And performing cold-hot separation, index creation and role separation on the updated service data in sequence to obtain the service data to be inquired.
Specifically, the server marks the updated service data according to a preset time node to obtain hot node data and cold node data, wherein the hot node data corresponds to a preset target time period, the cold node data corresponds to a time period other than the target time period, and the target time period is a query time period determined according to preset service requirements; index creation is carried out on the hot node data and the cold node data to obtain the node data with the created index; acquiring role configuration information corresponding to a search engine Elasticissearch cluster, wherein the role configuration information comprises roles, nodes corresponding to the roles and node configuration information of the nodes; and sending the node data with the created index to the node corresponding to the role according to the role and the node configuration information to obtain the service data to be inquired.
The target time period is a query time period determined according to a preset service requirement, for example: the preset business requirement is the query business requirement of the problem express, the problem express is the express with the condition in express logistics, the data volume of the problem express is large, the main business data query request is concentrated in the latest period R established by the order, and R is the target period.
Node labels, for example: hot node labeling is performed by bin/elastic search-E node.name ═ hot node-ecluster.name ═ get _ time-E path.data ═ hot _ data-E node.attr. my _ node _ type ═ hot node labeling, cold node labeling is performed by bin/elastic search-E node.name ═ cold node-E path.data ═ cold _ data-E node.
Role configuration information and service data to be queried, for example: the roles are a master role, an ingest role and a data role, the nodes and node configuration information corresponding to the master role are servers W1 (resource configuration 30%), W2 (resource configuration 50%) and W3 (resource configuration 20%) configured to normal performance, the nodes and node configuration information corresponding to the ingest role are servers T1 (resource configuration 40%), T2 (resource configuration 30%) and T3 (resource configuration 30%) configured to good performance, the nodes and node configuration information corresponding to the data role are servers O1 (resource configuration 50%), O2 (resource configuration 30%) and O3 (resource configuration 20%) configured to good storage performance, the node data for scheduling in the indexed node data is respectively sent to W1, W2 and W3 according to 30%, 50% and 20%, the node data for data preprocessing in the indexed node data is respectively sent to T1, 30% and 30% respectively, And T2 and T3, sending node data for scheduling in 50%, 30% and 20% of the node data with the created index to O1, O2 and O3, respectively.
Specifically, the server judges whether the updated service data is in a preset service data life cycle; if the updated service data is not in the preset service data life cycle, determining the updated service data as historical data; creating an index cluster on a node corresponding to the hot node data to obtain corresponding first node data, wherein the first node data comprises a node index and the generation time of the service data; sending the historical data to a node corresponding to the cold node data to obtain target cold node data; acquiring the current moment, and determining a node index to be migrated in the node index according to the generation moment and the current moment; migrating the node index to be migrated to a node corresponding to the target cold node data through a preset interface to obtain corresponding second node data; the first node data and the second node data are determined as the node data of which the index has been created.
For example: through "index. In the form of a "hot" which, creating an index cluster on a node corresponding to the hot node data to obtain corresponding first node data, wherein the size of each index is within 20G or within 50G, sending the historical data to a node corresponding to the cold node data to obtain updated cold node data, judging the corresponding time in the first node data to obtain the result that the time 2020.02.01 corresponding to the node data J is not in the target time period 2020.03, the index corresponding to the node data J is determined as the node index to be migrated, complaint-2020-02-01, through PUT compatibility-2020-03/_ settings { "index. And (2) migrating the node index to be migrated complaint-2020-02-01 to a node corresponding to the target cold node data by 'co ld', so as to obtain second node data, wherein the first node data and the second node data are the node data of the created index.
The server may determine the node index to be migrated in the node index according to the time difference between the generation time and the current time and the size of the preset threshold, for example: the generation time of the service data is 2020.02.29, the current time is 2020.03.09, the time difference between the generation time and the current time is 9 days, if the preset threshold is 5 days, the node index corresponding to 2020.02.29 is determined as the node index to be migrated, and if the preset threshold is 10 days, the node index corresponding to 2020.02.29 is not migrated. The preset threshold may be set according to the target time period, that is, the preset threshold may be a duration corresponding to the target time period.
The node index is used for inquiring the node corresponding to the hot node data and the service data corresponding to the node. The preset service data life cycle is a life cycle of express logistics service data, for example: creating express bills, managing warehouses, transporting logistics vehicles, delivering and processing after-sales service. Judging whether the updated service data is in a preset service data life cycle, and if the updated service data is not in the preset service data life cycle, determining the updated service data as historical data, for example: the last stage of the preset service data life cycle is an after-sales service processing stage, and if the state of the flow node for updating the service data is after the after-sales service processing stage (i.e. not in the preset service data life cycle), the updated service data is historical data. The historical data is updated service data that is not within a preset service data life cycle, and may be updated service data whose current flow node is a final flow node and which is in an end state, for example: and the final flow node is customer service arbitration management, the current flow node for updating the service data is customer service arbitration management, the running state is a finishing state, and the customer service arbitration management is a stage outside the preset logistics service data life cycle, so that the updated service data is historical data.
If the updated service data is in the preset service data life cycle, determining the updated service data as real-time data, for example: one of the preset service data life cycles is an express delivery management stage, and the process node state of the updated service data is in express delivery (namely in the preset service data life cycle), so that the updated service data is real-time data. The real-time data is updated service data in a preset service data life cycle, and may be that a current process node of the updated service data is a final process node or not, and the current process node is in operation, for example: and the final flow node is customer service arbitration management, if the current flow node for updating the service data is express cargo storage management (before customer service arbitration management) and the running state is in progress, the updated service data is real-time data, and if the current flow node for updating the service data is customer service arbitration management and the running state is in progress, the updated service data is real-time data.
Specifically, the server can also perform index creation on the hot node data and the cold node data to obtain an initial index; carrying out fragmentation setting on the initial index according to the preset number of main fragments to obtain the fragmented initial index; performing retrieval demand analysis, aggregation analysis and word segmentation demand analysis on the fragmented initial index to obtain a candidate index; performing attribute setting and type setting on the candidate index to obtain a target index; and determining the hot node data created with the target index and the cold node data created with the target index as the node data of the created index.
For the fragmentation setting of the initial index according to the preset number of main fragments, the maximum number of fragments of the initial index on one node can be set through the parameter total _ guard _ per _ node, so that the phenomenon that a small number of nodes are distributed to the fragmented initial index fragments in a centralized manner and load imbalance is caused is avoided.
Performing retrieval demand analysis, aggregation analysis and word segmentation demand analysis on the fragmented initial index to obtain a candidate index; and performing attribute setting and type setting on the candidate indexes, for example: if the fields in the fragmented initial index do not need to be retrieved, determining the fields as candidate indexes, and setting the index attribute of the fields of the candidate indexes as failure false, so that the reverse index of the fields does not need to be constructed, and the data writing performance is improved; if the character string field in the fragmented initial index does not need to be participled, determining the character string field as a candidate index, and setting the field type of the candidate index as a keyword so as to improve the efficiency of creating and retrieving the index.
204. And receiving a service data query request, and querying service data to be queried in the Elasticissearch cluster of the search engine according to the service data query request to obtain target query data.
Specifically, the server receives a service data query request and extracts query time information in the service data query request; analyzing nodes corresponding to the hot node data and the cold node data according to the query time information and the role configuration information to obtain corresponding target nodes; and searching the node data of which the index is created in the target node to obtain target query data corresponding to the service data query request.
The query time information and the role configuration information have corresponding analysis sequences, that is, nodes corresponding to the hot node data and the cold node data are analyzed according to the query time information to obtain corresponding query nodes, and a target node in the query nodes is determined according to the role configuration information, for example: the query time information in the service data query request is 2019.10.5 (time corresponding to hot node data), the role configuration information is that nodes corresponding to roles in the hot node data are Y1 and Y2, the nodes corresponding to the hot node data and the cold node data are analyzed through 2019.10.5 to obtain nodes corresponding to the hot node data (namely query nodes), the nodes corresponding to the hot node data comprise Y1, Y2 and Y3, the nodes corresponding to the hot node data are analyzed according to the role configuration information to determine corresponding target nodes Y1 and Y2, the indexed node data in Y1 and Y2 are retrieved, and the target query data corresponding to the service data query request is obtained.
In determining a target node in the query node according to the role configuration information, the server may determine the target node in the nodes through the resource occupation ratio of each node in the search engine elastic search cluster and the role configuration information, for example: the obtained query nodes are Q1, Q2 and Q3, the resource proportion of each node in the search engine Elastic search cluster is respectively 90% of the resource proportion of Q1, 50% of the resource proportion of Q2 and 20% of the resource proportion of Q3, the role configuration information is that the nodes corresponding to the roles in the hot-node data are Q1 and Q3, the node resources of Q1 are relatively more utilized and the query efficiency is influenced, and therefore the target node in the query node is determined to be Q3 according to the role configuration information.
It should be noted that, through the above operations, load balancing of the Elasticsearch cluster is ensured, resources are reasonably utilized, query is facilitated, and query efficiency is improved.
205. And recording and statistically analyzing the business data query request and the target query data to obtain statistical analysis data, wherein the statistical analysis data is used for optimizing cold-hot separation of updated business data.
The server records each business data query request and each target query data to obtain recorded data, performs statistical analysis on the recorded data to obtain statistical analysis data, wherein the statistical analysis data can be query time periods with more query times, and adjusts time periods of preset time nodes for cold and hot separation according to the query time periods to improve the accuracy of cold and hot data separation of updated business data, so that the query accuracy of the business data to be queried is improved.
The server can determine the final target time period according to the superposition of the query time periods and the time periods corresponding to the query items, for example, the query time periods are K1, K2 and K3, the query items are L1, L2 and L3, the time period corresponding to L3 is K2, the query item corresponding to K2 is L3, the K2 is the final target time period, and the K2 and the queried times are statistical analysis data.
In the embodiment of the invention, on the basis of meeting the requirements on storage and operation performance of a large amount of logistics business data and the requirements on multi-condition query, the synchronization and read-write separation of the logistics business data are realized, and the problem that the conventional logistics business data processing scheme cannot meet the business object processing logic requirements of a large amount of logistics business data is solved, the cold-hot separation of the updated business data is optimized through statistical analysis data, the accuracy of the cold-hot data separation of the updated business data is improved, and the query accuracy of the business data to be queried is improved.
In the above description of the service data processing method based on the Elasticsearch in the embodiment of the present invention, the following description of the service data processing apparatus based on the Elasticsearch in the embodiment of the present invention refers to fig. 3, and an embodiment of the service data processing apparatus based on the Elasticsearch in the embodiment of the present invention includes:
the updating module 301 is configured to receive a service data updating request, and update service data in a preset relational database according to the service data updating request to obtain updated service data, where the service data is logistics service data;
a writing module 302, configured to write the update service data into a preset search engine Elasticsearch cluster through a preset open source data synchronization tool Canal;
the processing module 303 is configured to perform cold-hot separation, index creation, and role separation on the updated service data in sequence to obtain service data to be queried;
the query module 304 is configured to receive a service data query request, and query the service data to be queried in the Elasticsearch cluster of the search engine according to the service data query request, so as to obtain target query data.
The function implementation of each module in the service data processing apparatus based on the Elasticsearch corresponds to each step in the embodiment of the service data processing method based on the Elasticsearch, and the function and implementation process thereof are not described in detail herein.
In the embodiment of the invention, the logistics service data in the preset relational database are updated, the updated logistics service data are synchronized to the search engine Elasticissearch cluster through the open source data synchronization tool Canal, and the search engine Elasticissearch cluster is queried, so that the requirements on storage and operation performance of a large amount of logistics service data and the requirements on multi-condition query are met, the logistics service data are synchronized and read-write separated, and the problem that the conventional logistics service data processing scheme cannot meet the service object processing logic requirements of a large amount of logistics service data is solved.
Referring to fig. 4, another embodiment of the service data processing apparatus based on an elastic search in the embodiment of the present invention includes:
the updating module 301 is configured to receive a service data updating request, and update service data in a preset relational database according to the service data updating request to obtain updated service data, where the service data is logistics service data;
a writing module 302, configured to write the update service data into a preset search engine Elasticsearch cluster through a preset open source data synchronization tool Canal;
the processing module 303 is configured to perform cold-hot separation, index creation, and role separation on the updated service data in sequence to obtain service data to be queried;
the query module 304 is configured to receive a service data query request, and query service data to be queried in the search engine elastic search cluster according to the service data query request to obtain target query data;
and the record counting module 305 is configured to record and perform statistical analysis on the service data query request and the target query data to obtain statistical analysis data, where the statistical analysis data is used to optimize the cold-hot separation of updated service data.
Optionally, the processing module 303 includes:
a node marking unit 3031, configured to mark a node on the updated service data according to a preset time node to obtain hot node data and cold node data, where the hot node data corresponds to a preset target time period, and the cold node data corresponds to a time period other than the target time period, where the target time period is a query time period determined according to a preset service requirement;
an index creating unit 3032, configured to create an index for the hot node data and the cold node data, to obtain node data with the created index;
an obtaining unit 3033, configured to obtain role configuration information corresponding to the search engine Elasticsearch cluster, where the role configuration information includes roles, nodes corresponding to the roles, and node configuration information of the nodes;
a sending unit 3034, configured to send the node data with the created index to a node corresponding to the role according to the role and the node configuration information, so as to obtain service data to be queried.
Optionally, the index creating unit 3032 may further be specifically configured to:
judging whether the updated service data is in a preset service data life cycle;
if the updated service data is not in the preset service data life cycle, determining the updated service data as historical data;
creating an index cluster on a node corresponding to the hot node data to obtain corresponding first node data, wherein the first node data comprises a node index and the generation time of the service data;
sending the historical data to a node corresponding to the cold node data to obtain target cold node data;
acquiring the current moment, and determining a node index to be migrated in the node index according to the generation moment and the current moment;
migrating the node index to be migrated to a node corresponding to the target cold node data through a preset interface to obtain corresponding second node data;
the first node data and the second node data are determined as the node data of which the index has been created.
Optionally, the index creating unit 3032 may further be specifically configured to:
index creation is carried out on the hot node data and the cold node data to obtain an initial index;
carrying out fragmentation setting on the initial index according to the preset number of main fragments to obtain the fragmented initial index;
performing retrieval demand analysis, aggregation analysis and word segmentation demand analysis on the fragmented initial index to obtain a candidate index;
performing attribute setting and type setting on the candidate index to obtain a target index;
and determining the hot node data created with the target index and the cold node data created with the target index as the node data of the created index.
Optionally, the query module 304 may be further specifically configured to:
receiving a service data query request, and extracting query time information in the service data query request;
analyzing nodes corresponding to the hot node data and the cold node data according to the query time information and the role configuration information to obtain corresponding target nodes;
and searching the node data of which the index is created in the target node to obtain target query data corresponding to the service data query request.
Optionally, the writing module 302 may be further specifically configured to:
extracting the updated service data through a preset open source data synchronization tool (cancer), and sending the updated service data to a preset cache region to obtain cached updated service data;
and according to a synchronization configuration file in an open source data synchronization tool, synchronizing the cached updated service data to a preset search engine Elasticissearch cluster.
The functional implementation of each module and each unit in the service data processing apparatus based on the Elasticsearch corresponds to each step in the embodiment of the service data processing method based on the Elasticsearch, and the functions and implementation processes thereof are not described in detail herein.
In the embodiment of the invention, on the basis of meeting the requirements on storage and operation performance of a large amount of logistics business data and the requirements on multi-condition query, the synchronization and read-write separation of the logistics business data are realized, and the problem that the conventional logistics business data processing scheme cannot meet the business object processing logic requirements of a large amount of logistics business data is solved, the cold-hot separation of the updated business data is optimized through statistical analysis data, the accuracy of the cold-hot data separation of the updated business data is improved, and the query accuracy of the business data to be queried is improved.
Fig. 3 and fig. 4 describe the service data processing apparatus based on the Elasticsearch in the embodiment of the present invention in detail from the perspective of the modular functional entity, and the service data processing apparatus based on the Elasticsearch in the embodiment of the present invention is described in detail from the perspective of hardware processing.
Fig. 5 is a schematic structural diagram of an Elasticsearch-based business data processing apparatus according to an embodiment of the present invention, where the Elasticsearch-based business data processing apparatus 500 may generate relatively large differences due to different configurations or performances, and may include one or more processors (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) storing an application 533 or data 532. Memory 520 and storage media 530 may be, among other things, transient or persistent storage. The program stored in the storage medium 530 may include one or more modules (not shown), and each module may include a series of instructions operating on the Elasticsearch-based business data processing apparatus 500. Further, the processor 510 may be configured to communicate with the storage medium 530, and execute a series of instruction operations in the storage medium 530 on the Elasticsearch-based business data processing apparatus 500.
The Elasticissearch-based service data processing device 500 may further comprise one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input-output interfaces 560, and/or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, L inux, FreeBSD, etc. it will be understood by those skilled in the art that the Elasticissearch-based service data processing device architecture shown in FIG. 5 does not constitute a limitation of the Elasticissearch-based service data processing device, and may comprise more or fewer components than those shown, or some components in combination, or a different arrangement of components.
The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and may also be a volatile computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the instructions cause the computer to execute the steps of the Elasticsearch-based business data processing method.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A service data processing method based on an elastic search is characterized by comprising the following steps:
receiving a service data updating request, and updating service data in a preset relational database according to the service data updating request to obtain updated service data, wherein the service data is logistics service data;
writing the updated service data into a preset search engine Elasticissearch cluster through a preset open source data synchronization tool (cancer);
performing cold-hot separation, index creation and role separation on the updated service data in sequence to obtain service data to be inquired;
and receiving a service data query request, and querying service data to be queried in the search engine Elasticissearch cluster according to the service data query request to obtain target query data.
2. The method for processing service data based on elastic search of claim 1, wherein the performing cold-hot separation, index creation and role separation on the updated service data in sequence to obtain the service data to be queried comprises:
carrying out node marking on the updated service data according to a preset time node to obtain hot node data and cold node data, wherein the hot node data corresponds to a preset target time period, the cold node data corresponds to a time period other than the target time period, and the target time period is a query time period determined according to a preset service requirement;
index creation is carried out on the hot node data and the cold node data, and node data with created indexes are obtained;
acquiring role configuration information corresponding to the search engine Elasticissearch cluster, wherein the role configuration information comprises roles, nodes corresponding to the roles and node configuration information of the nodes;
and sending the node data with the created index to a node corresponding to the role according to the role and the node configuration information to obtain service data to be inquired.
3. The transit data processing method based on Elasticsearch according to claim 2, wherein the index creation of the hot node data and the cold node data to obtain the node data with created index includes:
judging whether the updated service data is in a preset service data life cycle;
if the updated service data is not in the preset service data life cycle, determining the updated service data as historical data;
creating an index cluster on a node corresponding to the hot node data to obtain corresponding first node data, wherein the first node data comprises a node index and the generation time of service data;
sending the historical data to a node corresponding to the cold node data to obtain target cold node data;
acquiring the current moment, and determining a node index to be migrated in the node indexes according to the generation moment and the current moment;
migrating the node index to be migrated to a node corresponding to the target cold node data through a preset interface to obtain corresponding second node data;
determining the first node data and the second node data as indexed node data.
4. The transit data processing method based on Elasticsearch according to claim 2, wherein the index creation of the hot node data and the cold node data to obtain the node data with created index includes:
index creation is carried out on the hot node data and the cold node data to obtain an initial index;
carrying out fragmentation setting on the initial index according to a preset number of main fragments to obtain a fragmented initial index;
performing retrieval demand analysis, aggregation analysis and word segmentation demand analysis on the fragmented initial index to obtain a candidate index;
performing attribute setting and type setting on the candidate index to obtain a target index;
and determining the hot node data created with the target index and the cold node data created with the target index as the node data of the created index.
5. The method for processing Elasticissearch-based service data according to claim 2, wherein the step of receiving a service data query request and querying the service data to be queried in the search engine Elasticissearch cluster according to the service data query request to obtain target query data comprises the steps of:
receiving a service data query request, and extracting query time information in the service data query request;
analyzing nodes corresponding to the hot node data and the cold node data according to the query time information and the role configuration information to obtain corresponding target nodes;
and retrieving the node data of which the index is created in the target node to obtain target query data corresponding to the service data query request.
6. The method for processing Elasticissearch-based business data according to claim 1, wherein said writing the updated business data into a preset search engine Elasticissearch cluster through a preset open source data synchronization tool, Canal, comprises:
extracting the updated service data through a preset open source data synchronization tool (cancer), and sending the updated service data to a preset cache region to obtain cached updated service data;
and synchronizing the cached updated service data to a preset search engine Elasticissearch cluster according to a synchronization configuration file in the open source data synchronization tool Canal.
7. The method for processing service data based on an Elasticsearch according to any of claims 1 to 6, wherein after receiving a service data query request and querying service data to be queried in the search engine Elasticsearch cluster according to the service data query request, the method further comprises:
and recording and statistically analyzing the business data query request and the target query data to obtain statistical analysis data, wherein the statistical analysis data is used for optimizing cold-hot separation of the updated business data.
8. An elastic search based service data processing device, characterized in that the elastic search based service data processing device comprises:
the updating module is used for receiving a service data updating request and updating service data in a preset relational database according to the service data updating request to obtain updated service data, wherein the service data is logistics service data;
the writing module is used for writing the updated service data into a preset search engine Elasticissearch cluster through a preset open source data synchronization tool (cancer);
the processing module is used for sequentially carrying out cold-hot separation, index creation and role separation on the updated service data to obtain service data to be inquired;
and the query module is used for receiving a service data query request and querying the service data to be queried in the search engine Elasticissearch cluster according to the service data query request to obtain target query data.
9. An elastic search based service data processing device, characterized in that the elastic search based service data processing device comprises: a memory having instructions stored therein and at least one processor, the memory and the at least one processor interconnected by a line;
the at least one processor calls the instruction in the memory to cause the elastic search based business data processing device to execute the elastic search based business data processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the Elasticsearch-based traffic data processing method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010358044.7A CN111460023B (en) | 2020-04-29 | 2020-04-29 | Method, device, equipment and storage medium for processing service data based on elastic search |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010358044.7A CN111460023B (en) | 2020-04-29 | 2020-04-29 | Method, device, equipment and storage medium for processing service data based on elastic search |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111460023A true CN111460023A (en) | 2020-07-28 |
| CN111460023B CN111460023B (en) | 2023-06-09 |
Family
ID=71682651
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010358044.7A Active CN111460023B (en) | 2020-04-29 | 2020-04-29 | Method, device, equipment and storage medium for processing service data based on elastic search |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111460023B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112181993A (en) * | 2020-10-27 | 2021-01-05 | 广州市网星信息技术有限公司 | Service data query method, device, server and storage medium |
| CN112181987A (en) * | 2020-10-12 | 2021-01-05 | 嘉联支付有限公司 | Non-time sequence data processing method |
| CN112445854A (en) * | 2020-11-25 | 2021-03-05 | 平安普惠企业管理有限公司 | Multi-source business data real-time processing method and device, terminal and storage medium |
| CN112527911A (en) * | 2020-12-29 | 2021-03-19 | 上海销氪信息科技有限公司 | Data storage method, device, equipment and medium |
| CN112800104A (en) * | 2020-12-08 | 2021-05-14 | 江苏苏宁云计算有限公司 | Method and device for optimizing ES query request link |
| CN112883252A (en) * | 2021-02-05 | 2021-06-01 | 成都新希望金融信息有限公司 | Service query method, device, computer equipment and readable storage medium |
| CN112925783A (en) * | 2021-03-26 | 2021-06-08 | 北京金山云网络技术有限公司 | Service data processing method and device, electronic equipment and storage medium |
| CN113282618A (en) * | 2021-06-18 | 2021-08-20 | 福建天晴数码有限公司 | Optimization scheme and system for retrieval of active clusters of Elasticissearch |
| CN113763099A (en) * | 2020-12-29 | 2021-12-07 | 京东城市(北京)数字科技有限公司 | Data searching method, device, equipment and storage medium |
| CN114564485A (en) * | 2022-04-28 | 2022-05-31 | 深圳竹云科技股份有限公司 | User data processing method based on Elastic Search |
| CN116089545A (en) * | 2023-04-07 | 2023-05-09 | 云筑信息科技(成都)有限公司 | Method for collecting storage medium change data into data warehouse |
| CN116401259A (en) * | 2023-06-08 | 2023-07-07 | 北京江融信科技有限公司 | Automatic pre-creation index method and system for elastic search database |
| CN117093367A (en) * | 2023-08-22 | 2023-11-21 | 广州今之港教育咨询有限公司 | Service data processing method, device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018095351A1 (en) * | 2016-11-28 | 2018-05-31 | 中兴通讯股份有限公司 | Method and device for search processing |
| CN108681593A (en) * | 2018-05-16 | 2018-10-19 | 青岛海信移动通信技术股份有限公司 | Service data retrieval method and device |
| WO2018233364A1 (en) * | 2017-06-19 | 2018-12-27 | 华为技术有限公司 | Index updating method and system, and related device |
| CN110569302A (en) * | 2019-08-16 | 2019-12-13 | 苏宁云计算有限公司 | method and device for physical isolation of distributed cluster based on lucene |
-
2020
- 2020-04-29 CN CN202010358044.7A patent/CN111460023B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018095351A1 (en) * | 2016-11-28 | 2018-05-31 | 中兴通讯股份有限公司 | Method and device for search processing |
| WO2018233364A1 (en) * | 2017-06-19 | 2018-12-27 | 华为技术有限公司 | Index updating method and system, and related device |
| CN108681593A (en) * | 2018-05-16 | 2018-10-19 | 青岛海信移动通信技术股份有限公司 | Service data retrieval method and device |
| CN110569302A (en) * | 2019-08-16 | 2019-12-13 | 苏宁云计算有限公司 | method and device for physical isolation of distributed cluster based on lucene |
Non-Patent Citations (1)
| Title |
|---|
| 陈伦跃;殷峰;: "基于搜索引擎的慢查询优化系统" * |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112181987A (en) * | 2020-10-12 | 2021-01-05 | 嘉联支付有限公司 | Non-time sequence data processing method |
| CN112181993A (en) * | 2020-10-27 | 2021-01-05 | 广州市网星信息技术有限公司 | Service data query method, device, server and storage medium |
| CN112445854A (en) * | 2020-11-25 | 2021-03-05 | 平安普惠企业管理有限公司 | Multi-source business data real-time processing method and device, terminal and storage medium |
| CN112445854B (en) * | 2020-11-25 | 2024-05-03 | 北京品域互联科技有限公司 | Multi-source service data real-time processing method, device, terminal and storage medium |
| CN112800104A (en) * | 2020-12-08 | 2021-05-14 | 江苏苏宁云计算有限公司 | Method and device for optimizing ES query request link |
| CN113763099A (en) * | 2020-12-29 | 2021-12-07 | 京东城市(北京)数字科技有限公司 | Data searching method, device, equipment and storage medium |
| CN112527911A (en) * | 2020-12-29 | 2021-03-19 | 上海销氪信息科技有限公司 | Data storage method, device, equipment and medium |
| CN112883252A (en) * | 2021-02-05 | 2021-06-01 | 成都新希望金融信息有限公司 | Service query method, device, computer equipment and readable storage medium |
| CN112925783A (en) * | 2021-03-26 | 2021-06-08 | 北京金山云网络技术有限公司 | Service data processing method and device, electronic equipment and storage medium |
| CN113282618A (en) * | 2021-06-18 | 2021-08-20 | 福建天晴数码有限公司 | Optimization scheme and system for retrieval of active clusters of Elasticissearch |
| CN114564485A (en) * | 2022-04-28 | 2022-05-31 | 深圳竹云科技股份有限公司 | User data processing method based on Elastic Search |
| CN116089545A (en) * | 2023-04-07 | 2023-05-09 | 云筑信息科技(成都)有限公司 | Method for collecting storage medium change data into data warehouse |
| CN116089545B (en) * | 2023-04-07 | 2023-08-22 | 云筑信息科技(成都)有限公司 | Method for collecting storage medium change data into data warehouse |
| CN116401259A (en) * | 2023-06-08 | 2023-07-07 | 北京江融信科技有限公司 | Automatic pre-creation index method and system for elastic search database |
| CN116401259B (en) * | 2023-06-08 | 2023-08-22 | 北京江融信科技有限公司 | Automatic pre-creation index method and system for elastic search database |
| CN117093367A (en) * | 2023-08-22 | 2023-11-21 | 广州今之港教育咨询有限公司 | Service data processing method, device and storage medium |
| CN117093367B (en) * | 2023-08-22 | 2024-04-09 | 广州今之港教育咨询有限公司 | Service data processing method, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111460023B (en) | 2023-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111460023A (en) | Service data processing method, device, equipment and storage medium based on elastic search | |
| CN111460024B (en) | Real-time service system based on elastic search | |
| CN111459985B (en) | Identification information processing method and device | |
| US7624118B2 (en) | Data processing over very large databases | |
| EP2924594B1 (en) | Data encoding and corresponding data structure in a column-store database | |
| US8280917B1 (en) | Batching content management operations to facilitate efficient database interactions | |
| CN108073696B (en) | GIS application method based on distributed memory database | |
| US9477772B2 (en) | Storing and retrieving objects on a computer network in a distributed database | |
| CN106066895A (en) | A kind of intelligent inquiry system | |
| WO2015043391A1 (en) | Data synchronization for remote and local databases | |
| CN110096509A (en) | Realize that historical data draws the system and method for storage of linked list modeling processing under big data environment | |
| CN106933836A (en) | A kind of date storage method and system based on point table | |
| CN102779138A (en) | Hard disk access method of real time data | |
| CN110109948A (en) | Data query method, computer equipment and computer readable storage medium | |
| Santos et al. | Optimizing data warehouse loading procedures for enabling useful-time data warehousing | |
| CN106161193B (en) | Mail processing method, device and system | |
| CN113806446A (en) | Rapid retrieval method for mass data of big data | |
| US20200293507A1 (en) | Auto Unload | |
| CN110689386A (en) | Electronic bill issuing method and system | |
| CN102597969A (en) | Database management device using key-value store with attributes, and key-value-store structure caching-device therefor | |
| CN114003660A (en) | Method and device for efficiently synchronizing real-time data to click House based on flash | |
| CN117331513B (en) | Data reduction method and system based on Hadoop architecture | |
| CN117009439B (en) | Data processing method, device, electronic equipment and storage medium | |
| Krompaß et al. | Data staging for OLAP-and OLTP-applications on RFID data | |
| CN117349327A (en) | Memory data acquisition optimization method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |