CN114328364B - Data transmission method, device and equipment in full text search engine and storage medium - Google Patents
Data transmission method, device and equipment in full text search engine and storage medium Download PDFInfo
- Publication number
- CN114328364B CN114328364B CN202111447679.5A CN202111447679A CN114328364B CN 114328364 B CN114328364 B CN 114328364B CN 202111447679 A CN202111447679 A CN 202111447679A CN 114328364 B CN114328364 B CN 114328364B
- Authority
- CN
- China
- Prior art keywords
- data
- node
- search engine
- determining
- cache pool
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 230000005540 biological transmission Effects 0.000 title claims abstract description 61
- 238000007726 management method Methods 0.000 claims description 41
- 230000001360 synchronised effect Effects 0.000 claims description 40
- 239000012634 fragment Substances 0.000 claims description 15
- 238000004422 calculation algorithm Methods 0.000 claims description 13
- 238000012163 sequencing technique Methods 0.000 claims description 12
- 238000005516 engineering process Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 7
- 230000002688 persistence Effects 0.000 claims description 7
- 238000013500 data storage Methods 0.000 claims description 5
- 230000004931 aggregating effect Effects 0.000 claims description 3
- 230000004044 response Effects 0.000 claims 2
- 238000010586 diagram Methods 0.000 description 6
- 230000005291 magnetic effect Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 238000005192 partition Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000005294 ferromagnetic effect Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data transmission method and device in a full text search engine, electronic equipment and a computer readable storage medium, wherein the full text search engine comprises a management node, a coordination node and a data node, and the data node comprises a main data node for storing data and a secondary data node for storing data copies; the method comprises the following steps: determining a source node and a destination node of data transmission, and determining target data to be transmitted in the source node; transmitting the target data to a second HCA card in the destination node through the first HCA card of the source node; the first HCA card and the second HCA card are both HCA cards that use the RDMA protocol. According to the application, the first HCA card and the second HCA card are respectively arranged in the source node and the destination node, and the target data to be transmitted is transmitted between the first HCA card and the second HCA card through the RDMA protocol, so that the data transmission efficiency in the full-text search engine is improved.
Description
Technical Field
The present application relates to the field of computer technology, and more particularly, to a data transmission method and apparatus in a full text search engine, an electronic device, and a computer readable storage medium.
Background
Because of the development of Internet and Internet of things applications, how to retrieve mass data becomes a technical development challenge, and the traditional relational database can meet the retrieval requirement of the relational data, but the current growth of unstructured data exceeds structured relational data, so as to solve the full-text retrieval requirement of the mass data and the unstructured data, and how to solve the high-efficiency and rapid full-text retrieval engine (simply called retrieval engine) technology becomes a current technical hotspot.
Currently, data synchronization transmission between clusters is realized between data nodes of an Elastic Search (ES) cluster based on a mainstream TCP (transmission control protocol ) protocol. The main data slicing receives data transmitted by a client, the data is written into a slicing data buffer area, the data in the buffer reaches flush capacity or time reaches flush time slice requirements, and the data in the buffer is flushed into a transaction log file. After the main data fragments are written into the log file, the ES process calls a data synchronization interface to prepare for synchronizing data to a copy node, the specific process is shown in figure 1, the main data fragments copy the fragment cache data to an operating system kernel mode cache, the operating system kernel cache copies the fragment cache data to a local network card cache, the local network card cache transfers the fragment cache to a copy node network card cache, the copy node copies the received data to an operating system cache area from the network card cache, the ES copies the data to an ES user mode cache from the kernel cache after receiving an operating system instruction, and the data is written into the copy node log file.
It can be seen that the data needs to be replicated a minimum of three times inside each source node: and copying the ES buffer to an operating system kernel buffer and then copying the ES buffer to a network card buffer. Three replications also need to be done within the target node: the network card is cached, copied to the operating system cache and then copied to the ES cache. The data transmission between the ES data nodes uses TCP protocol, TCP establishes the link and needs to carry on three times of handshakes, the efficiency of data transmission.
Therefore, how to improve the data transmission efficiency in the full text search engine is a technical problem that needs to be solved by those skilled in the art.
Disclosure of Invention
The application aims to provide a data transmission method and device in a full-text search engine, electronic equipment and a computer readable storage medium, and improves the data transmission efficiency in the full-text search engine.
In order to achieve the above object, the present application provides a data transmission method in a full text search engine, where the full text search engine includes a management node, a coordination node, and a data node, and the data node includes a primary data node for storing data and a secondary data node for storing a copy of the data;
the method comprises the following steps:
determining a source node and a destination node of data transmission, and determining target data to be transmitted in the source node;
Transmitting the target data to a second HCA card in the destination node through a first HCA card of the source node; wherein the first HCA card and the second HCA card are both HCA cards using RDMA protocol.
The determining the source node and the destination node of the data transmission, and determining the target data to be transmitted in the source node includes:
determining the main data node as a source node and the auxiliary data node as a destination node;
determining data acquired by the main data node through an RDMA interface as target data; the main data node acquires the target data through an RDMA interface and stores the target data.
The full text search engine comprises a plurality of management nodes, the source node and the destination node for determining data transmission, and the target data to be transmitted are determined in the source node, and the full text search engine comprises:
determining a first management node as a source node and a second management node as a destination node;
and determining the metadata information in the first management node as target data to be transmitted.
Wherein after the target data is sent to the second HCA card in the destination node by the first HCA card of the source node, the method further comprises:
Applying for a buffer space for the target data in a buffer pool of the target node, and storing the target data in the buffer space;
and storing the target data in the cache space into a disk Translog file of the destination node by utilizing RDMA technology.
Before storing the target data into the cache space, the method further comprises:
judging whether the data volume in the cache pool is larger than a preset proportion of the total capacity of the cache pool or not;
if yes, clearing part of data in the cache pool;
and if not, executing the step of storing the target data into the cache space.
Wherein the clearing the part of the data in the cache pool includes:
calculating the weight of all pages in the cache pool according to an LRU algorithm, and sequencing all pages in the cache pool according to the order of the weight from large to small;
and clearing pages in the cache pool from back to front based on the sequencing result until the data volume of the cleared pages is greater than or equal to the data volume of the target data.
The coordination node is used for receiving a search request of a client and determining a target data node corresponding to the search request;
The determining the source node and the destination node of the data transmission, and determining the target data to be transmitted in the source node, includes:
determining the target data node as a source node and the coordination node as a target node;
and determining a search result corresponding to the search request in the target data node as target data to be transmitted.
In order to achieve the above object, the present application provides a data transmission device in a full text search engine, where the full text search engine includes a management node, a coordination node, and a data node, and the data node includes a primary data node for storing data and a secondary data node for storing a copy of the data;
the device comprises:
the system comprises a determining module, a transmitting module and a receiving module, wherein the determining module is used for determining a source node and a destination node of data transmission and determining target data to be transmitted in the source node;
a transmitting module, configured to transmit, by using a first HCA card of the source node, the target data to a second HCA card in the destination node; wherein the first HCA card and the second HCA card are both HCA cards using RDMA protocol.
To achieve the above object, the present application provides an electronic device including:
A memory for storing a computer program;
and a processor for implementing the steps of the data transmission method in the full text search engine when executing the computer program.
To achieve the above object, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a data transmission method in a full text search engine as described above.
According to the scheme, the data transmission method in the full-text search engine comprises a management node, a coordination node and a data node, wherein the data node comprises a main data node for storing data and a secondary data node for storing data copies; the method comprises the following steps: determining a source node and a destination node of data transmission, and determining target data to be transmitted in the source node; transmitting the target data to a second HCA card in the destination node through a first HCA card of the source node; wherein the first HCA card and the second HCA card are both HCA cards using RDMA protocol.
According to the data transmission method in the full-text search engine, the first HCA (Host Channel Adapter ) card and the second HCA card are respectively arranged in the source node and the destination node, target data to be transmitted is transmitted between the first HCA card and the second HCA card through the RDMA protocol, and compared with the TCP protocol, three-way handshake is not needed, so that the data transmission efficiency is improved. The application also discloses a data transmission device in the full text search engine, an electronic device and a computer readable storage medium, and the technical effects can be realized.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application as claimed.
Drawings
In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. The accompanying drawings are included to provide a further understanding of the disclosure, and are incorporated in and constitute a part of this specification, illustrate the disclosure and together with the description serve to explain, but do not limit the disclosure. In the drawings:
FIG. 1 is a schematic diagram of a data synchronization method provided in the related art;
FIG. 2 is a flow chart illustrating a method of data transmission in a full text search engine according to an exemplary embodiment;
FIG. 3 is a block diagram of a full text search engine cluster, shown in accordance with an exemplary embodiment;
FIG. 4 is a flowchart of an RDMA according to an exemplary embodiment;
FIG. 5 is a block diagram of a data transmission device in a full text search engine, according to an exemplary embodiment;
fig. 6 is a block diagram of an electronic device, according to an example embodiment.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application. In addition, in the embodiments of the present application, "first", "second", etc. are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence.
The embodiment of the application discloses a data transmission method in a full-text search engine, which improves the data transmission efficiency in the full-text search engine.
Referring to fig. 2, a flowchart of a data transmission method in a full text search engine is shown according to an exemplary embodiment, as shown in fig. 2, including:
s101: determining a source node and a destination node of data transmission, and determining target data to be transmitted in the source node;
The embodiment is applied to a full-text search engine, and comprises a management node (Master node), a data node (Datanode node) and a coordination node (Coordinator node), wherein the management node is responsible for collecting and updating cluster metadata information, and metadata information synchronization is carried out among different management nodes. The data nodes are responsible for data storage and inquiry, the data nodes comprise main data nodes and auxiliary data nodes, the main data nodes are used for storing data, the auxiliary data nodes are used for storing data copies, and data stored between the main data nodes and the corresponding auxiliary data nodes are synchronous. The coordination node is responsible for receiving a client search request, establishing a search request route, aggregating the intermediate sets of search results of all data nodes participating in the search, sequencing and merging the intermediate sets into a global result, and returning the global result to the client.
The present embodiment employs RDMA (Remote Direct Memory Access ) for data transfer between a source node and a destination node. RDMA is a new direct memory access technology that allows computers to directly access the memory of other computers without requiring processing by a processor. In implementation, RDMA is actually a remote memory direct high-speed access technology with fully optimized intelligent network card and software architecture, and the aim of high-performance remote direct data access is achieved by solidifying RDMA protocol on hardware and supporting two approaches of Zero-copy and Kernel bypass. The advantages of using RDMA are as follows: zero copy (Zero-copy): the application can perform data transfer directly, and without involving the network software stack, the data can be sent directly to or received from the buffer without being copied to the network layer. Kernel bypass (Kernel bypass): the application program can directly execute data transmission in the user mode without context switching between the kernel mode and the user mode. CPU intervention is not required (No CPU involvement): the application may access the remote host memory without consuming any CPU in the remote host, which can be read without the involvement of a process (or CPU) on the remote host, the cache of the remote host's CPU not being filled with the accessed memory content.
As a possible implementation manner, the determining a source node and a destination node of data transmission, and determining target data to be transmitted in the source node, includes: determining the main data node as a source node and the auxiliary data node as a destination node; determining data acquired by the main data node through an RDMA interface as target data; the main data node acquires the target data through an RDMA interface and stores the target data. In a specific implementation, an RDMA interface is added in the master data node for receiving target data and storing the target data. The primary data node needs to synchronize the target data to the secondary data node, that is, the source node in this embodiment is the primary data node, and the destination node is the secondary data node.
As another possible implementation manner, the full text search engine includes a plurality of management nodes, the determining a source node and a destination node of data transmission, and determining target data to be transmitted in the source node includes: determining a first management node as a source node and a second management node as a destination node; and determining the metadata information in the first management node as target data to be transmitted. In a specific implementation, the full-text search engine comprises a plurality of management nodes, wherein the management nodes each store full metadata information, the full-text search engine comprises cluster-level metadata, index-level metadata, fragment-level metadata and the like, the cluster-level metadata comprises cluster configuration data, cluster template data and the like, the index-level metadata comprises index settings, fragment mapping, a word divider, aliases and the like, and the fragment-level metadata comprises version numbers, index identification codes, main fragment data and the like. The metadata information needs to be synchronized among the plurality of management nodes, that is, the target data in the embodiment is metadata information, the source node is a first management node, and the destination node is a second management node. Specifically, an RDMA interface is added in the management nodes, in the synchronization process of metadata information, whether the cluster is configured with a network card supporting an RDMA protocol is judged, if the network card supporting the RDMA protocol is configured, the RDMA interface is called to synchronize the metadata information among a plurality of management nodes, otherwise, the management nodes adopt a TCP protocol to communicate and synchronize the metadata information.
As yet another possible implementation manner, the coordination node is configured to receive a search request of a client, and determine a target data node corresponding to the search request; the determining the source node and the destination node of the data transmission, and determining the target data to be transmitted in the source node, includes: determining the target data node as a source node and the coordination node as a target node; and determining a search result corresponding to the search request in the target data node as target data to be transmitted. It can be understood that the coordination node is configured to receive a client search request, determine a target data node corresponding to the search request, query the target data node for target data, and return the target data to the client. It can be seen that the target data node needs to transmit the target data to the coordination node, that is, the source node in this embodiment is the target data node, and the destination node is the coordination node. In the implementation, an RDMA interface is added in a coordination node, in the transmission process of target data, whether a cluster is configured with a network card supporting an RDMA protocol is judged, if the network card supporting the RDMA protocol is configured, the RDMA interface is called to transmit the target data, and otherwise, the TCP protocol is adopted to communicate and transmit the target data.
S102: transmitting the target data to a second HCA card in the destination node through a first HCA card of the source node; wherein the first HCA card and the second HCA card are both HCA cards using RDMA protocol.
In this embodiment, a first HCA card and a second HCA card are respectively set in a source node and a destination node, and destination data is transmitted between the first HCA card and the second HCA card through an RDMA protocol, and the destination node stores the data to be synchronized after receiving the data. By using RDMA communication technology, the data can be directly transmitted from the source node user mode to the destination node user mode RDMA cache, context switching of the kernel mode user mode is not needed, and data transmission efficiency is improved.
It may be appreciated that, if the embodiment is used for storing data synchronously between the primary data node and the secondary data node, or synchronizing metadata information between a plurality of management nodes, after the target data is sent to the second HCA card in the destination node by the first HCA card of the source node, the method further includes: applying for a buffer space for the target data in a buffer pool of the target node, and storing the target data in the buffer space; and storing the target data in the cache space into a disk Translog file of the destination node by utilizing RDMA technology. It should be noted that, both the source node and the destination node can use the DRAM memory as the cache, so that the read-write efficiency is higher. In a specific implementation, after the source node obtains the data to be synchronized, a buffer space is reserved for the data to be synchronized in a buffer pool of the source node, and the data to be synchronized is placed in the applied buffer space. Further, data persistence is performed, namely, target data is written into a local disk Translog file of the destination node. The RDMA technology can be utilized to directly write the data to be synchronized into the disk in the user mode, so that the data storage efficiency is improved.
As a preferred embodiment, before storing the target data in the cache space, the method further includes: judging whether the data volume in the cache pool is larger than a preset proportion of the total capacity of the cache pool or not; if yes, clearing part of data in the cache pool; and if not, executing the step of storing the target data into the cache space. In a specific implementation, a elimination adjustment algorithm is implemented on the cache pool, and the data amount in the cache pool is kept not to exceed a preset proportion of the total capacity of the cache pool. As a possible implementation manner, calculating weights of all pages in the cache pool according to an LRU (Least Recently Used ) algorithm, and sorting all pages in the cache pool according to the order of the weights from high to low; and clearing pages in the cache pool from back to front based on the sequencing result until the data volume of the cleared pages is greater than or equal to the data volume of the target data. In a specific implementation, the weights of the pages in the cache pool are calculated and ordered according to the LRU algorithm, if the data amount in the cache pool exceeds a preset proportion, for example 90%, of the total capacity of the cache pool, the pages with low weights in the cache pool are cleared, and the clear page selection calculation method comprises the following steps: and selecting N pages with the lowest page weights, wherein the capacity of the N pages is larger than or equal to the data quantity of the data to be synchronized.
It should be noted that, the main data node needs to store the target data after receiving the target data, and the specific storage mode is the same as the above-described mode, which is not described herein again. After the main data node stores the target data, the first HCA card sends the target data to a second HCA card in the auxiliary data node, and the auxiliary data node stores the target data in the mode described above.
According to the data transmission method in the full text search engine, the first HCA card and the second HCA card are respectively arranged in the source node and the destination node, target data to be transmitted are transmitted between the first HCA card and the second HCA card through the RDMA protocol, and compared with the TCP protocol, three-way handshake is not needed, so that the data synchronization efficiency is improved.
In the following, an application embodiment of the present application is described, and in particular, a full text search engine cluster is shown in fig. 3, and includes a management node (Master node), a plurality of data nodes (Datanode nodes) and a coordinating node (Coordinator node), where the management node does not participate in data storage and query, and the full text search cluster is a self-built election algorithm inside the management node for guaranteeing high availability (HA, high Availability).
The client writes a brief flow of data into the ES cluster: the client sends a write request to a Server0 node (coordination node) of the ES cluster, the coordination node obtains a value of 1 through a route calculation formula, then the current data is written to the ES main partition 1, the coordination node forwards the request to a node DataNode1 where the main partition 1 is located, the DataNode1 receives the request, distributes the data to a data segment, firstly writes the data into a cache of the ES, and writes the data in the cache into a disk according to a data persistence strategy. And the ES shards send the data to the DataNode2 and DataNode3 where the two data shard copies are located according to a copy algorithm, so that the ES shards on the two nodes complete the writing of the two copy shard data, and the conflict of the data is controlled through optimistic concurrency. Once all duplicate slices report success, node DataNode1 will report success to the coordinator node, which reports success to the client. The management node synchronously updates metadata such as data segments, index libraries, documents and the like in the retrieval. The client completes the data write request.
In implementations, a storage node for a copy of the target data may be determined based on metadata information for the full-text search engine cluster. The RDMA flow is as shown in FIG. 4, and mainly comprises the following steps:
step one: the data slicing 1 (main slicing) writes data to be synchronized into an RDMA memory cache pool, and the specific flow is as follows:
1.1: the search engine process applies for local cache for the data to be synchronized in the local memory cache pool;
1.2: the method comprises the steps that a full-text search engine process to which a data main fragment belongs calls an RDMA synchronous interface, and a server list to which a data fragment copy belongs is queried in search engine metadata;
1.3: according to the LRU algorithm, calculating the weight of pages in the cache pool and sequencing, if the number of the cache pool is 90% of the total capacity of the cache pool, clearing the pages with low weight in the cache pool, and clearing the page selection calculation method: selecting N pages with lowest page weights, wherein the capacity of the N pages is larger than or equal to the data quantity of the data to be synchronized;
1.4: putting data to be synchronized into a page of a cache pool, and keeping the data capacity in the cache pool not to exceed 90% of the total capacity;
1.5: the search engine calls an operating system data persistence interface and writes the data to be synchronized into a local disk Translog file;
Step two: the search engine calls an RDMA interface to send the data to be synchronized to the nodes where all the partitioned copy search engines are located, each node search engine of the partitioned copy executes the operation of 1.1-1.5, the memory page scheduling of the local cache pool is carried out, and the data transmitted by the main partition through the RDMA interface is stored in the memory cache pool of the copy engine;
step three: and after receiving the data synchronization completion information fed back by the engines to which all the slicing copies belong, the master slicing engine completes the data synchronization.
The RDMA communication technology is synchronously used by the full text retrieval engine data, and the following steps can be realized: the data is directly transmitted to the RDMA cache of the user mode of the duplicate data node from the user mode of the master slice node, and context switching of the user mode of the kernel mode is not needed, so that the data transmission efficiency is improved.
In addition, an RDMA-based data retrieval interface is added in a searching intermediate data retrieval method of a query module of the coordination node, RDMA protocol support judgment is added, and if the RDMA protocol is supported in configuration, the coordination node fetches data from a data node cache through the RDMA protocol; otherwise, the data is obtained according to the original method; adding RDMA protocol support judgment in a data return interface of a data node, and if the RDMA protocol is supported in configuration, placing a query result in an RDMA buffer by the data node; otherwise, the product is stored according to the original method.
The following describes a data transmission device in a full text search engine according to an embodiment of the present application, and the data transmission device in the full text search engine described below and the data transmission method in the full text search engine described above may be referred to each other.
Referring to fig. 5, a block diagram of a data transmission device in a full text search engine is shown according to an exemplary embodiment, as shown in fig. 5, including:
a determining module 501, configured to determine a source node and a destination node of data transmission, and determine target data to be transmitted in the source node;
a sending module 502, configured to send, by using a first HCA card of the source node, the target data to a second HCA card in the destination node; wherein the first HCA card and the second HCA card are both HCA cards using RDMA protocol.
According to the data transmission method in the full text search engine, the first HCA card and the second HCA card are respectively arranged in the source node and the destination node, target data to be transmitted are transmitted between the first HCA card and the second HCA card through the RDMA protocol, and compared with the TCP protocol, three-way handshake is not needed, so that the data synchronization efficiency is improved.
On the basis of the foregoing embodiment, as a preferred implementation manner, the determining module 501 specifically determines the primary data node as a source node, determines the secondary data node as a destination node, and determines data acquired by the primary data node through an RDMA interface as target data; the main data node acquires the target data through an RDMA interface and stores the target data.
On the basis of the foregoing embodiment, as a preferred implementation manner, the full text search engine includes a plurality of management nodes, and the determining module 501 specifically determines a first management node as a source node, determines a second management node as a destination node, and determines metadata information in the first management node as target data to be transmitted.
On the basis of the above embodiment, as a preferred implementation manner, the method further includes:
an application module, configured to apply for a buffer space in a buffer pool of the destination node for the target data;
the first storage module is used for storing the target data into the cache space;
and the second storage module is used for storing the target data in the cache space into a disk Translog file of the destination node by utilizing an RDMA technology.
On the basis of the above embodiment, as a preferred implementation manner, the method further includes:
the judging module is used for judging whether the data volume in the cache pool is larger than the preset proportion of the total capacity of the cache pool; if yes, starting the workflow of the clearing module; if not, starting the workflow of the first storage module;
and the clearing module is used for clearing part of data in the cache pool.
Based on the foregoing embodiment, as a preferred implementation manner, the cleaning module includes:
the ordering unit is used for calculating the weights of all the pages in the cache pool according to the LRU algorithm and ordering all the pages in the cache pool according to the order of the weights from large to small;
and the clearing unit is used for clearing pages in the cache pool from back to front based on the sequencing result until the data volume of the cleared pages is greater than or equal to the data volume of the target data.
On the basis of the foregoing embodiment, as a preferred implementation manner, the coordination node is configured to receive a search request of a client, and determine a target data node corresponding to the search request;
the determining module 501 specifically determines the target data node as a source node, determines the coordinating node as a destination node, and determines a search result corresponding to the search request in the target data node as target data to be transmitted.
Based on the hardware implementation of the program modules, and in order to implement the method according to the embodiment of the present application, the embodiment of the present application further provides an electronic device, and fig. 6 is a block diagram of an electronic device according to an exemplary embodiment, and as shown in fig. 6, the electronic device includes:
a communication interface 1 capable of information interaction with other devices such as network devices and the like;
and the processor 2 is connected with the communication interface 1 to realize information interaction with other devices and is used for executing the data synchronization method provided by one or more technical schemes when running the computer program. And the computer program is stored on the memory 3.
Of course, in practice, the various components in the electronic device are coupled together by a bus system 4. It will be appreciated that the bus system 4 is used to enable connected communications between these components. The bus system 4 comprises, in addition to a data bus, a power bus, a control bus and a status signal bus. But for clarity of illustration the various buses are labeled as bus system 4 in fig. 6.
The memory 3 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program for operating on an electronic device.
It will be appreciated that the memory 3 may be either volatile memory or nonvolatile memory, and may include both volatile and nonvolatile memory. Wherein the nonvolatile Memory may be Read Only Memory (ROM), programmable Read Only Memory (PROM, programmable Read-Only Memory), erasable programmable Read Only Memory (EPROM, erasable Programmable Read-Only Memory), electrically erasable programmable Read Only Memory (EEPROM, electrically Erasable Programmable Read-Only Memory), magnetic random access Memory (FRAM, ferromagnetic random access Memory), flash Memory (Flash Memory), magnetic surface Memory, optical disk, or compact disk Read Only Memory (CD-ROM, compact Disc Read-Only Memory); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be random access memory (RAM, random Access Memory), which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available, such as static random access memory (SRAM, static Random Access Memory), synchronous static random access memory (SSRAM, synchronous Static Random Access Memory), dynamic random access memory (DRAM, dynamic Random Access Memory), synchronous dynamic random access memory (SDRAM, synchronous Dynamic Random Access Memory), double data rate synchronous dynamic random access memory (ddr SDRAM, double Data Rate Synchronous Dynamic Random Access Memory), enhanced synchronous dynamic random access memory (ESDRAM, enhanced Synchronous Dynamic Random Access Memory), synchronous link dynamic random access memory (SLDRAM, syncLink Dynamic Random Access Memory), direct memory bus random access memory (DRRAM, direct Rambus Random Access Memory). The memory 2 described in the embodiments of the present application is intended to comprise, without being limited to, these and any other suitable types of memory.
The method disclosed in the above embodiment of the present application may be applied to the processor 2 or implemented by the processor 2. The processor 2 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in the processor 2 or by instructions in the form of software. The processor 2 described above may be a general purpose processor, DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The processor 2 may implement or perform the methods, steps and logic blocks disclosed in embodiments of the present application. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiment of the application can be directly embodied in the hardware of the decoding processor or can be implemented by combining hardware and software modules in the decoding processor. The software modules may be located in a storage medium in the memory 3 and the processor 2 reads the program in the memory 3 to perform the steps of the method described above in connection with its hardware.
The corresponding flow in each method of the embodiments of the present application is implemented when the processor 2 executes the program, and for brevity, will not be described in detail herein.
In an exemplary embodiment, the present application also provides a storage medium, i.e. a computer storage medium, in particular a computer readable storage medium, for example comprising a memory 3 storing a computer program executable by the processor 2 for performing the steps of the method described above. The computer readable storage medium may be FRAM, ROM, PROM, EPROM, EEPROM, flash Memory, magnetic surface Memory, optical disk, or CD-ROM.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware associated with program instructions, where the foregoing program may be stored in a computer readable storage medium, and when executed, the program performs steps including the above method embodiments; and the aforementioned storage medium includes: a removable storage device, ROM, RAM, magnetic or optical disk, or other medium capable of storing program code.
Alternatively, the above-described integrated units of the present application may be stored in a computer-readable storage medium if implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied in essence or a part contributing to the prior art in the form of a software product stored in a storage medium, including several instructions for causing an electronic device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a removable storage device, ROM, RAM, magnetic or optical disk, or other medium capable of storing program code.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. The data transmission method in the full text search engine is characterized in that the full text search engine comprises a management node, a coordination node and a data node, wherein the data node comprises a main data node for storing data and a secondary data node for storing data copies; the management nodes are responsible for collecting and updating cluster metadata information, and metadata information synchronization is carried out among different management nodes; the data nodes are responsible for data storage and inquiry, and the data stored between the main data nodes and the corresponding auxiliary data nodes are synchronous; the coordination node is responsible for receiving a client search request, establishing a search request route, aggregating the intermediate sets of search results of all data nodes participating in the search, sequencing and merging the intermediate sets into a global result, and returning the global result to the client; the coordination node is further configured to receive a write request of the client, determine a first master slice in the full text search engine through a routing calculation formula, forward the write request to a first data node where the first master slice is located, so that the first data node distributes data to a data segment, writes the data into a cache of the full text search engine, writes the data in the cache to a disk according to a data persistence policy, and then causes the slice of the full text search engine to send the data to a second data contact where a copy of the data slice is located according to a copy algorithm and control the conflict of the data through optimistic concurrency, and receive report success information sent by the first data contact after the copy of the data contact reports success, and report success to the client;
The method comprises the following steps:
determining a source node and a destination node of data transmission, and determining target data to be transmitted in the source node;
transmitting the target data to a second HCA card in the destination node through a first HCA card of the source node; wherein the first HCA card and the second HCA card are both HCA cards using RDMA protocol;
wherein, in response to determining a storage node for a copy of the target data based on metadata information of the full-text search engine cluster, the flow of RDMA comprises:
step one: the data main sharding writes the data to be synchronized into an RDMA memory cache pool, and the specific flow is as follows:
1.1: the search engine process applies for local cache for the data to be synchronized in a local memory cache pool;
1.2: the full text search engine process of the data main fragment calls an RDMA synchronous interface, and the server list of the data fragment copy is queried in the metadata of the search engine;
1.3: according to the LRU algorithm, calculating the weight of pages in the cache pool and sequencing, if the number of the cache pool exceeds the preset proportion of the total capacity of the cache pool, clearing the pages with low weight in the cache pool, and clearing the page selection calculation method: selecting N pages with lowest page weights, wherein the capacity of the N pages is larger than or equal to the data quantity of the data to be synchronized;
1.4: the data to be synchronized is put into a page of a cache pool, and the data capacity in the cache pool is kept not to exceed the preset proportion of the total capacity;
1.5: the search engine calls an operating system data persistence interface and writes the data to be synchronized into a local disk Translog file;
step two: the search engine calls an RDMA interface to send the data to be synchronized to nodes where all the partitioned copy search engines are located, each node search engine of the partitioned copy executes the operation of 1.1-1.5, memory page scheduling of a local cache pool is carried out, and the data transmitted by the data main partitioned through the RDMA interface is stored in the memory cache pool of the copy engine;
step three: and after receiving the data synchronization completion information fed back by the engines to which all the slicing copies belong, the master slicing engine completes the data synchronization.
2. The data transmission method according to claim 1, wherein the determining a source node and a destination node of data transmission and determining target data to be transmitted in the source node includes:
determining the main data node as a source node and the auxiliary data node as a destination node;
determining data acquired by the main data node through an RDMA interface as target data; the main data node acquires the target data through an RDMA interface and stores the target data.
3. The data transmission method according to claim 1, wherein the full text search engine includes a plurality of management nodes, the determining a source node and a destination node of data transmission, and determining target data to be transmitted in the source node includes:
determining a first management node as a source node and a second management node as a destination node;
and determining the metadata information in the first management node as target data to be transmitted.
4. A data transmission method according to claim 2 or 3, wherein after the target data is transmitted to the second HCA card in the destination node by the first HCA card of the source node, further comprising:
applying for a buffer space for the target data in a buffer pool of the target node, and storing the target data in the buffer space;
and storing the target data in the cache space into a disk Translog file of the destination node by utilizing RDMA technology.
5. The data transmission method according to claim 4, further comprising, before storing the target data in the buffer space:
judging whether the data volume in the cache pool is larger than a preset proportion of the total capacity of the cache pool or not;
If yes, clearing part of data in the cache pool;
and if not, executing the step of storing the target data into the cache space.
6. The method of claim 5, wherein the clearing the portion of the data in the buffer pool comprises:
calculating the weight of all pages in the cache pool according to an LRU algorithm, and sequencing all pages in the cache pool according to the order of the weight from large to small;
and clearing pages in the cache pool from back to front based on the sequencing result until the data volume of the cleared pages is greater than or equal to the data volume of the target data.
7. The data transmission method according to claim 1, wherein the coordination node is configured to receive a search request from a client, and determine a target data node corresponding to the search request;
the determining the source node and the destination node of the data transmission, and determining the target data to be transmitted in the source node, includes:
determining the target data node as a source node and the coordination node as a target node;
and determining a search result corresponding to the search request in the target data node as target data to be transmitted.
8. A data transmission device in a full text search engine, wherein the full text search engine comprises a management node, a coordination node and a data node, and the data node comprises a main data node for storing data and a secondary data node for storing data copies; the management nodes are responsible for collecting and updating cluster metadata information, and metadata information synchronization is carried out among different management nodes; the data nodes are responsible for data storage and inquiry, and the data stored between the main data nodes and the corresponding auxiliary data nodes are synchronous; the coordination node is responsible for receiving a client search request, establishing a search request route, aggregating the intermediate sets of search results of all data nodes participating in the search, sequencing and merging the intermediate sets into a global result, and returning the global result to the client; the coordination node is further configured to receive a write request of the client, determine a first master slice in the full text search engine through a routing calculation formula, forward the write request to a first data node where the first master slice is located, so that the first data node distributes data to a data segment, writes the data into a cache of the full text search engine, writes the data in the cache to a disk according to a data persistence policy, and then causes the slice of the full text search engine to send the data to a second data contact where a copy of the data slice is located according to a copy algorithm and control the conflict of the data through optimistic concurrency, and receive report success information sent by the first data contact after the copy of the data contact reports success, and report success to the client;
The device comprises:
the system comprises a determining module, a transmitting module and a receiving module, wherein the determining module is used for determining a source node and a destination node of data transmission and determining target data to be transmitted in the source node;
a transmitting module, configured to transmit, by using a first HCA card of the source node, the target data to a second HCA card in the destination node; wherein the first HCA card and the second HCA card are both HCA cards using RDMA protocol;
wherein, in response to determining a storage node for a copy of the target data based on metadata information for a full-text search engine cluster, the apparatus-executed RDMA flow comprises:
step one: the data main sharding writes the data to be synchronized into an RDMA memory cache pool, and the specific flow is as follows:
1.1: the search engine process applies for local cache for the data to be synchronized in a local memory cache pool;
1.2: the full text search engine process of the data main fragment calls an RDMA synchronous interface, and the server list of the data fragment copy is queried in the metadata of the search engine;
1.3: according to the LRU algorithm, calculating the weight of pages in the cache pool and sequencing, if the number of the cache pool exceeds the preset proportion of the total capacity of the cache pool, clearing the pages with low weight in the cache pool, and clearing the page selection calculation method: selecting N pages with lowest page weights, wherein the capacity of the N pages is larger than or equal to the data quantity of the data to be synchronized;
1.4: the data to be synchronized is put into a page of a cache pool, and the data capacity in the cache pool is kept not to exceed the preset proportion of the total capacity;
1.5: the search engine calls an operating system data persistence interface and writes the data to be synchronized into a local disk Translog file;
step two: the search engine calls an RDMA interface to send the data to be synchronized to nodes where all the partitioned copy search engines are located, each node search engine of the partitioned copy executes the operation of 1.1-1.5, memory page scheduling of a local cache pool is carried out, and data transmitted by the data main partitioned through the RDMA interface is stored in a memory cache pool of the copy engine;
step three: and after receiving the data synchronization completion information fed back by the engines to which all the slicing copies belong, the master slicing engine completes the data synchronization.
9. An electronic device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data transmission method in a full text search engine as claimed in any one of claims 1 to 7 when executing said computer program.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the data transmission method in a full text search engine as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111447679.5A CN114328364B (en) | 2021-11-30 | 2021-11-30 | Data transmission method, device and equipment in full text search engine and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111447679.5A CN114328364B (en) | 2021-11-30 | 2021-11-30 | Data transmission method, device and equipment in full text search engine and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114328364A CN114328364A (en) | 2022-04-12 |
| CN114328364B true CN114328364B (en) | 2023-11-07 |
Family
ID=81049121
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111447679.5A Active CN114328364B (en) | 2021-11-30 | 2021-11-30 | Data transmission method, device and equipment in full text search engine and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114328364B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113395359A (en) * | 2021-08-17 | 2021-09-14 | 苏州浪潮智能科技有限公司 | File currency cluster data transmission method and system based on remote direct memory access |
-
2021
- 2021-11-30 CN CN202111447679.5A patent/CN114328364B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113395359A (en) * | 2021-08-17 | 2021-09-14 | 苏州浪潮智能科技有限公司 | File currency cluster data transmission method and system based on remote direct memory access |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114328364A (en) | 2022-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10958752B2 (en) | Providing access to managed content | |
| CN109213772B (en) | Data storage method and NVMe storage system | |
| US6883068B2 (en) | Methods and apparatus for implementing a chche replacement scheme | |
| US20230087447A1 (en) | Data migration method and device | |
| US9569742B2 (en) | Reducing costs related to use of networks based on pricing heterogeneity | |
| US20200081867A1 (en) | Independent evictions from datastore accelerator fleet nodes | |
| CN103530387A (en) | Improved method aimed at small files of HDFS | |
| CN104679898A (en) | Big data access method | |
| US20190347167A1 (en) | Primary Node-Standby Node Data Transmission Method, Control Node, and Database System | |
| US20150254320A1 (en) | Using colocation hints to facilitate accessing a distributed data storage system | |
| CN101382953A (en) | Interface system for accessing file system in user space and file reading and writing method | |
| CN111386522A (en) | Multi-region multi-master replication of database tables | |
| US20190278757A1 (en) | Distributed Database Management System with Dynamically Split B-Tree Indexes | |
| CN112162846B (en) | Transaction processing method, device and computer readable storage medium | |
| US20200133869A1 (en) | Method, electronic device and computer program product for data storage | |
| WO2015039569A1 (en) | Copy storage device and copy storage method | |
| CN108647266A (en) | A kind of isomeric data is quickly distributed storage, exchange method | |
| CN111309266A (en) | Distributed storage metadata system log optimization system and method based on ceph | |
| CN114328364B (en) | Data transmission method, device and equipment in full text search engine and storage medium | |
| CN105740166B (en) | Cache reading and reading processing method and device | |
| JPH07239808A (en) | Distributed data managing system | |
| CN102055795A (en) | Distributed file system metadata management method | |
| Pabmanabhan et al. | DREAM: a data replication technique for real-time mobile ad-hoc network databases | |
| CN105786608A (en) | Remote deduplication migration method and system for virtual machine | |
| CN117539915B (en) | Data processing method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |