WO2021189253A1

WO2021189253A1 - Data transmission method and apparatus

Info

Publication number: WO2021189253A1
Application number: PCT/CN2020/080896
Authority: WO
Inventors: 郭子亮
Original assignee: 深圳市欢太科技有限公司; Oppo广东移动通信有限公司
Priority date: 2020-03-24
Filing date: 2020-03-24
Publication date: 2021-09-30
Also published as: CN115087968A

Abstract

The embodiments of the present application disclose a data transmission method and apparatus, being applied to a storage server in a data transmission system. The data transmission system further comprises a terminal device, a gateway server, a transit server, and at least one elastic search server cluster. Storing search request data in a message queue for reading by the transit server can improve the availability of the elastic search server cluster, avoiding unstability or unavailability of the entire elastic search server cluster caused by large amount of data written into the search request data; in addition, as the search request data is stored in the message queue, the trajectory of the message can be traced back for each piece of search request data, and when the elastic search server cluster is anomalous, the search request data may also be re-read from the message queue, so as to perform fault recovery.

Description

Data transmission method and device

Technical field

This application relates to the field of electronic technology, in particular to a data transmission method and device.

Background technique

At present, the user’s terminal equipment uses the representational state transfer-ful application programming interface (restful api) provided by the elastic search (Es) engine or the client software development kit (client software development kit, client SDK interface), directly insert or update the search request data into the index of the elastic search server. When writing search request data to the elastic search server, you can use the put interface to submit multiple document data at a time, and use the bulk interface to submit one document data at a time.

Using the existing method to access the elastic search server, the submission speed and data volume of the search request data completely depend on the frequency of the terminal device calling the interface and the total number of data passed in each request (using the bulk interface). In the case that the elastic search server cluster has fewer resources available, if the user’s terminal device write speed is too high, there is a high probability that the elastic search server cluster will become unstable, or even cause the elastic search server cluster to be unavailable, thereby reducing usage The availability of data writing or search services provided by the elastic search server cluster is difficult to guarantee the availability of the elastic search server in the elastic search server cluster.

Summary of the invention

The embodiments of the present application provide a data transmission method and device, which can improve the availability of an elastic search server cluster, and avoid that the entire elastic search server cluster is unstable or unavailable due to the large amount of data written in the search request data.

In the first aspect, an embodiment of the present application provides a data transmission method, which is applied to a storage server in a data transmission system. The data transmission system further includes a terminal device, a gateway server, a transit server, and at least one elastic search server cluster. Methods include:

Receiving a search request message sent by the gateway server, the search request message carrying search request data, the search request message is sent by the terminal device to the gateway server, and then the gateway server A message generated according to the search request data;

Writing the search request message into a message queue;

Receive a subscription request sent by the transit server, send the search request message to the transit server according to the subscription request, and instruct the transit server to send the search request message to the at least one elastic search server cluster The first elastic search server in, the first elastic search server is any server in the at least one elastic search server cluster.

In a second aspect, an embodiment of the present application provides a data transmission method, which is applied to a transit server in a data transmission system. The data transmission system further includes a terminal device, a gateway server, a storage server, and at least one elastic search server cluster. Methods include:

Send a subscription request to the storage server; and read a search request message from the message queue of the storage server, the search request message carries search request data, and the search request message is sent by the terminal device to the search Request data to the gateway server, and then the gateway server generates the search request message according to the search request data and writes it to the message queue of the storage server;

The search request message is sent to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.

In a third aspect, an embodiment of the present application provides a data transmission device, which is applied to a storage server in a data transmission system, and the data transmission system further includes a terminal device, a gateway server, a transit server, and at least one elastic search server cluster, so The data transmission device includes:

The receiving unit is configured to receive a search request message sent by the gateway server, the search request message carries search request data, and the search request message is the terminal device sending the search request data to the gateway server, and then A message generated by the gateway server according to the search request data;

A writing unit for writing the search request message into the message queue;

The receiving unit is further configured to receive a subscription request sent by the transit server;

The first sending unit is configured to send the search request message to the transit server according to the subscription request, and instruct the transit server to send the search request message to the first in the at least one elastic search server cluster An elastic search server, and the first elastic search server is any server in the at least one elastic search server cluster.

In a fourth aspect, an embodiment of the present application provides a data transmission device, which is applied to a transit server in a data transmission system, and the data transmission system further includes a terminal device, a gateway server, a storage server, and at least one elastic search server cluster, so The data transmission device includes:

The second sending unit is configured to send a subscription request to the storage server;

The reading unit is configured to read a search request message from the message queue of the storage server, the search request message carries search request data, and the search request message is sent by the terminal device to the search request data. The gateway server, then the gateway server generates the search request message according to the search request data and writes it into the message queue of the storage server;

The second sending unit is further configured to send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is the at least one elastic search server cluster Any server in.

In a fifth aspect, an embodiment of the present application provides a storage server, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and are configured by the above Executed by the processor, and the foregoing program includes instructions for executing the steps in the first aspect of the embodiments of the present application.

In a sixth aspect, an embodiment of the present application provides a transit server, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and are configured by the above Executed by the processor, and the foregoing program includes instructions for executing the steps in the second aspect of the embodiments of the present application.

In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the above-mentioned computer-readable storage medium stores a computer program for electronic data exchange, wherein the above-mentioned computer program enables a computer to execute Some or all of the steps described in one aspect or the second aspect.

In an eighth aspect, the embodiments of the present application provide a computer program product, wherein the above-mentioned computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the above-mentioned computer program is operable to cause a computer to execute as implemented in this application. Examples include part or all of the steps described in the first aspect or the second aspect. The computer program product may be a software installation package.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1A is a schematic structural diagram of a data transmission system for implementing a data transmission method according to an embodiment of the present application;

FIG. 1B is a schematic flowchart of a data transmission method disclosed in an embodiment of the present application;

FIG. 1C is a schematic diagram of a demonstration of writing the search request message into a message queue according to an embodiment of the present application;

FIG. 1D is a schematic diagram illustrating a mapping relationship between a message collection name and an elastic search server cluster and index name provided by an embodiment of the present application;

FIG. 2 is a schematic flowchart of another data transmission method disclosed in an embodiment of the present application;

FIG. 3 is a schematic flowchart of another data transmission method disclosed in an embodiment of the present application;

4A is a schematic structural diagram of another storage server disclosed in an embodiment of the present application;

4B is a schematic structural diagram of another transit server disclosed in an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a data transmission device disclosed in an embodiment of the present application;

6A is a schematic structural diagram of another data transmission device disclosed in an embodiment of the present application;

FIG. 6B is a modified structure of the data transmission device shown in FIG. 6A disclosed in an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the application, the technical solutions in the embodiments of the application will be clearly and completely described below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific sequence. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes unlisted steps or units, or optionally also includes Other steps or units inherent to these processes, methods, products or equipment.

The reference to "embodiments" herein means that a specific feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art clearly and implicitly understand that the embodiments described herein can be combined with other embodiments.

In order to facilitate a better understanding of the technical solutions described in the present invention, the technical terms involved in the embodiments of the present invention are explained below:

Elastic search (Es) server: The Es server is a search server based on the Lucene engine. It provides a full-text search engine with distributed multi-user capabilities. It is a highly available and scalable search engine framework.

Message queue (MQ): It is a "first in, first out" data organization in the basic data structure.

Presentation layer state transfer-ful application programming interface (representational state transfer-ful application programming interface, restful api): Presentation layer state transfer application programming interface, with a software construction style designed to provide web services.

Software development kit (SDK): Generally, it is a collection of development tools used by software engineers to create application software for specific software packages, software frameworks, hardware platforms, and operating systems.

The terminal devices involved in the embodiments of this application may include various handheld devices with wireless communication functions (such as mobile phones, tablet computers, etc.), vehicle-mounted devices, wearable devices, computing devices, or other processing devices connected to wireless modems, as well as various Forms of User Equipment (User Equipment, UE), Mobile Station (Mobile Station, MS), and so on.

The following describes the embodiments of the application in detail.

The data transmission method provided in the embodiments of the present application is applicable to a scenario where a user's terminal device accesses an elastic search server through search request data. Please refer to FIG. 1A, which is a schematic structural diagram of a data transmission system for implementing a data transmission method according to an embodiment of the present invention. The data transmission system may include a terminal device 1001, a gateway server 1002, a storage server 1003, a transit server 1004, and at least one elastic search server cluster. Each elastic search server cluster in the at least one elastic search server cluster includes multiple elastic search servers 1004, The terminal device 1001 is a client device used by the user. The gateway server 1002 is connected to the terminal device 1001. The terminal device 1001 does not directly send search request data to the elastic search server cluster, but sends the search request data to the gateway server 1002. The storage server 1003 is connected to the gateway server 1002. The gateway server 1002 writes the received search request data in the form of a search request message into the storage server 1003. The storage server 1003 writes the search request message into the message queue. The storage server 1003 also communicates with the transit server 1004. Connect, the transit server 1004 sends a subscription message to the storage server 1003, and reads the search request message from the storage server 1003, and then the transit server 1004 sends the read search request message to the elastic search server 1005 in the elastic search server cluster , So as to realize the user's terminal equipment to access the elastic search server cluster.

Please refer to FIG. 1B. FIG. 1B is a schematic flowchart of a data transmission method disclosed in an embodiment of the present application, which is applied to a storage server in a data transmission system. An elastic search server cluster, the data transmission method includes the following steps:

101. Receive a search request message sent by the gateway server, where the search request message carries search request data, and the search request message is sent by the terminal device to the gateway server, and then the search request message is sent by the terminal device to the gateway server. A message generated by the gateway server according to the search request data.

In the embodiment of this application, when a user accesses the elastic search server cluster through a terminal device, the user service of the terminal device can send search request data to the gateway server, and then the gateway server generates a search request message according to the search request data, and sends the search request message Sent to the storage server, so that the storage server can receive the search request message sent by the gateway server.

Among them, the terminal device can pre-register the user service. When registering the user service, the terminal device can obtain the elastic search server cluster information corresponding to the index in advance, so that the terminal device can send the elastic search server to the gateway server when it needs search access. Search request message for cluster information.

102. Write the search request message to a message queue.

The storage server can receive different search request messages sent by the gateway server, so that the storage server can sequentially write the search request messages into the message queue according to the order in which the search request messages are received.

Optionally, the search request data includes elastic search server cluster information and index name information to which the search request data belongs. In step 102, writing the search request message to a message queue may include the following steps:

11. Divide the search request message into message sets according to the elastic search server cluster information and the index name information;

12. Write the message set to the message queue;

13. Establish a mapping relationship between the name of the message set corresponding to the search request message and the name of the elastic search server cluster and the index.

[Correct 25.05.2020 according to Rule 91]
Please refer to FIG. 1C. FIG. 1C is a schematic diagram of writing the search request message into the message queue according to an embodiment of the application. The storage server sends the search request message to the storage server according to the index name and the elastic search server cluster information. Other received search request messages with the same index name and elastic search server cluster information are divided into the same message set, so that one message set in the message queue corresponds to one message set name, and all search request messages in each message set have With the same index name and elastic search server cluster information, the storage server can also establish a mapping relationship between the message set name corresponding to the search request message and the elastic search server cluster and index name, as shown in Figure 1D, which is implemented in this application The example provides a demonstration diagram of the mapping relationship between the name of the message collection and the elastic search server cluster and index name. Topic A, Topic B, Topic C, and Topic D are the names of the message collections, C1, C2, C3.. .It is an elastic search server cluster, V1, V2, V3... are index names. By establishing the mapping relationship, the transit server can read the search request message in the message queue at the same time. The name of the message collection, elastic search server cluster, and index name corresponding to the message collection to which it belongs.

103. Receive a subscription request sent by the transit server, send the search request message to the transit server according to the subscription request, and instruct the transit server to send the search request message to the at least one elastic search The first elastic search server in the server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.

Wherein, the above-mentioned transit server may be, for example, a sink server.

The storage server may receive a subscription request sent by the transit server, and the subscription request is used to instruct the storage server to send the search request message in the message queue to the transit server.

In specific implementation, the storage server can read the search request set according to the order of the message set in the message queue. Because the read data and write data in the message queue follow the "first in first out" rule, it can ensure that the message queue is read The order of the search request data and the write search request data is the same.

Considering that different users will access the elastic search server cluster through different multiple terminal devices, when the amount of search request data is large, part of the search request data can be stored through the message queue, which is specific to the current elastic search server cluster. By storing the search request data in the message queue and reading it by the transit server, the availability of the elastic search server cluster can be improved, and the large amount of data written in the search request data can be avoided to cause the entire elastic search server The cluster is unstable or unavailable. In addition, because the search request data is stored in the message queue, the message track can be traced back for each search request data. When the elastic search server cluster is abnormal, the search request data can be read again from the message queue. , So as to recover from the failure.

It can be seen that the data transmission method described in the embodiment of this application is applied to a storage server in a data transmission system. The data transmission system also includes a terminal device, a gateway server, a transit server, and at least one elastic search server cluster. The search request message sent by the server. The search request message carries the search request data. The search request message is a message that the terminal device sends the search request data to the gateway server, and then the gateway server generates the search request data according to the search request data; writes the search request message into the message queue ; Receive the subscription request sent by the transit server, send the search request message to the transit server according to the subscription request, and instruct the transit server to send the search request message to the first elastic search server in at least one elastic search server cluster, the first elastic search server It is any server in at least one elastic search server cluster. In this way, the search request data can be stored in the message queue and read by the transit server, which can improve the availability of the elastic search server cluster and avoid writing data due to search requests. The large amount of imported data makes the entire elastic search server cluster unstable or unavailable. In addition, since the search request data is stored in the message queue, the trajectory of the message can be traced back for each search request data. When the elastic search server cluster is abnormal, You can also read the search request data from the message queue again to recover from the failure.

Consistent with the above, FIG. 2 is a schematic flowchart of a data transmission method disclosed in an embodiment of the present application. The data transmission method is applied to a transit server in a data transmission system. The data transmission system further includes a terminal device, a gateway server, a storage server, and at least one elastic search server cluster. The data transmission method includes the following steps:

201. Send a subscription request to the storage server; and read a search request message from the message queue of the storage server, the search request message carries search request data, and the search request message is sent by the terminal device. The search request data is sent to the gateway server, and then the gateway server generates the search request message according to the search request data and writes it into the message queue of the storage server.

In specific implementation, when a user accesses the elastic search server cluster through a terminal device, the user service of the terminal device can send search request data to the gateway server, and then the gateway server generates a search request message according to the search request data, and sends the search request message to The storage server, and further, the server can write the search request message into the message queue, so that the transit server can send a subscription request to the storage server; and read the search request message from the message queue of the storage server.

Optionally, in the above step 201, reading the search request message from the message queue of the storage server may include the following steps:

According to the sequence of the message sets in the message queue, the message set is read from the message queue of the storage server, and the message set includes the search request message.

Wherein, the search request data includes elastic search server cluster information and index name information to which the search request data belongs. When the storage server writes the search request message into the message queue, it is specifically based on the elastic search server cluster information and all information. The index name information divides the search request message into message sets, and then writes the message sets into the message queue, so that the transit server can read from the message queue of the storage server according to the sequence of the message sets in the message queue The message collection.

Optionally, in the foregoing step 201, before reading the search request message from the message queue of the storage server, the following steps may be further included:

2011. Acquire health status data of all elastic search servers in the at least one elastic search server cluster, where the health status data includes at least one of the following health status sub-data: indexing rate, indexing time, memory size, and request rejection number, Wherein, each of the health status sub-data corresponds to a parameter threshold;

2012. Determine whether there is health state sub-data in the health state data that exceeds a corresponding parameter threshold, and if not, execute the operation of reading the search request message from the message queue of the storage server.

The transit server can obtain the current index rate, index time, memory size, and request rejections of all elastic search servers in at least one elastic search server cluster. The index rate refers to the average value of multiple index rates corresponding to all elastic search servers. , Index time consumption refers to the sum of multiple index time consumptions corresponding to all elastic search servers, memory size refers to the sum of multiple memory sizes corresponding to all elastic search servers, and the number of request rejections refers to multiple indexes corresponding to all elastic search servers. The sum of the number of rejected requests. The transit server may also preset the first parameter threshold corresponding to the index rate, the second parameter threshold corresponding to the time-consuming indexing, the third parameter threshold corresponding to the memory size, and the fourth parameter threshold corresponding to the number of request rejections. Then respectively determine whether the index rate, index time consumption, memory size, and request rejection number exceed the corresponding parameter threshold. If the index rate, index time consumption, memory size, and request rejection number do not exceed the corresponding parameter threshold, you can start from the storage The search request message is read from the message queue of the server. If there is at least one health status sub-data that exceeds the corresponding parameter threshold, the execution of the operation of reading the search request message from the message queue of the storage server can be suspended, so that Dynamically control the read rate of read search request messages to avoid the situation where the elastic search server cluster becomes unavailable due to excessive write requests.

202. Send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.

The transit server may send the read search request message to the first elastic search server in the at least one elastic search server cluster, where the first elastic search server may be any server in the at least one elastic search server cluster.

Optionally, the search request data includes elastic search server cluster information. In step 202, sending the search request message to the first elastic search server in the at least one elastic search server cluster may include the following steps:

21. Determine the first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

22. Send the search request message to the first elastic search server.

Wherein, the search request data may include the elastic search server cluster information to be sent by the search request data. Therefore, the technical secondary school server may determine the first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information. An elastic search server cluster is an elastic search server cluster corresponding to the elastic search server cluster information. Then, the transit server may send the search request message to the first elastic search server in the first elastic search server cluster, and the first elastic search server may be any server in the first elastic search server cluster.

Optionally, the search request data includes elastic search server cluster information and version information of the elastic search engine. In step 202, the search request message is sent to the first elastic search in the at least one elastic search server cluster. The server can include the following steps:

23. Determine the first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

24. Determine the elastic search server corresponding to the version information in the first elastic search server cluster;

25. Use the elastic search server corresponding to the version information as the first elastic search server, and send the search request message to the first elastic search server.

Among them, the gateway server can support multiple versions of the elastic search engine, and the search request data can include the version information of the elastic search engine. Furthermore, when the search request data is sent from the terminal device to the gateway server, and then through the storage server, it reaches the transit server and is transited. The server may determine the first elastic search server corresponding to the version information in the first elastic search server cluster according to the version information of the elastic search engine in the search request data. Therefore, the user service of the terminal device can operate the elastic search server clusters of different versions through the same interface, so that the cost can be reduced in the process of cluster upgrade and version switching later.

Optionally, the transit server includes an SDK interface or an API interface. In step 22 or step 25, sending the search request message to the first elastic search server may include the following steps:

A1. Send the search request message to the first elastic search server through the SDK interface; or,

A2. Send the search request message to the first elastic search server through the API interface.

Among them, the transit server may include the software development kit SDK interface or API interface, so that the transit server can send the search request message to the first elastic search server through the SDK interface, or send the search request message to the first elastic search server through the API interface server.

Optionally, if the transit server receives a feedback message that the first elastic search server cluster is out of the network abnormality, in this embodiment of the application, the following steps may be further included:

The search request message is sent to a second elastic search server in a second elastic search server cluster in the at least one elastic search server cluster, where the second elastic search server cluster and the first elastic search server cluster are located Different computer rooms.

Among them, the transit server can write the same search request message to different elastic search server clusters. If the first elastic search server cluster goes out of the network abnormally, the transit server can also send the search request message to the second elastic search server cluster. The second elastic search server. The second elastic search server can be any server in the second elastic search server cluster, so that it can ensure that the search request data sent by the user through the terminal device can be accessed successfully, and the elastic search server cluster is minimized. The unavailable situation arises.

It can be seen that the data transmission method described in the embodiment of this application is applied to a transit server in a data transmission system. The data transmission system also includes a terminal device, a gateway server, a storage server, and at least one elastic search server cluster. Send a subscription request to the storage server; and read the search request message from the message queue of the storage server. The search request message carries search request data. The search request message is sent by the terminal device to the gateway server, and then the gateway server The search request message is generated according to the search request data and then written into the message queue of the storage server; the search request message is sent to the first elastic search server in at least one elastic search server cluster, and the first elastic search server is at least one elastic search server cluster In this way, by storing the search request data in the message queue and reading it by the transit server, the availability of the elastic search server cluster can be improved, and the large amount of data written by the search request data can be avoided. As a result, the entire elastic search server cluster is unstable or unavailable. In addition, since the search request data is stored in the message queue, the trace of the message can be traced back for each search request data. When the elastic search server cluster is abnormal, it can also be retrieved from the message queue again. Read the search request data for fault recovery.

Consistent with the above, FIG. 3 is a schematic flowchart of a data transmission method disclosed in an embodiment of the present application. Applied to the data transmission system shown in FIG. 1A, the data transmission system may include a terminal device, a gateway server, a storage server, a transit server, and at least one elastic search server cluster. The data transmission method includes the following steps:

301. The terminal device sends search request data to a gateway server.

302. The gateway server generates a search request message according to the search request data.

303. The gateway server sends the search request message to a storage server.

304. The storage server writes the search request message into a message queue.

305. The transit server sends a subscription request to the storage server.

306. The transit server reads the search request message from the message queue of the storage server.

307. Send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.

For specific descriptions of the above steps 301 to 307, reference may be made to the corresponding descriptions of the data transmission method described in FIG. 1B and FIG. 2, which will not be repeated here.

It can be seen that the data transmission method described in the embodiment of the present application sends search request data to a gateway server through a terminal device, the gateway server generates a search request message according to the search request data, and the gateway server sends the search request message To the storage server, the storage server writes the search request message into a message queue, the transit server sends a subscription request to the storage server, the transit server reads the search request message from the message queue of the storage server, and The search request message is sent to a first elastic search server in the at least one elastic search server cluster, and the first elastic search server is any server in the at least one elastic search server cluster. The search request data is stored in the message queue and read by the transit server, which can improve the availability of the elastic search server cluster and avoid the large amount of data written in the search request data, which may make the entire elastic search server cluster unstable or unavailable In addition, because the search request data is stored in the message queue, the message track can be traced back for each search request data. When the elastic search server cluster is abnormal, the search request data can also be read from the message queue again for failure recovery.

Please refer to FIG. 4A. FIG. 4A is a schematic structural diagram of a storage server disclosed in an embodiment of the present application. As shown in the figure, the storage server includes a processor, a memory, a communication interface, and one or more programs. Or multiple programs are stored in the above-mentioned memory and are configured to be executed by the above-mentioned processor, and the above-mentioned programs include instructions for executing the following steps:

Writing the search request message into a message queue;

In a possible example, the search request data includes elastic search server cluster information and index name information to which the search request data belongs. In the aspect of writing the search request message to the message queue, the above program is specifically used for Follow the instructions for the following steps:

Dividing the search request message into message sets according to the elastic search server cluster information and the index name information;

Writing the message set into the message queue;

A mapping relationship between the name of the message set corresponding to the search request message and the name of the elastic search server cluster and the index is established.

Please refer to Figure 4B. Figure 4B is a schematic structural diagram of a transit server disclosed in an embodiment of the present application. As shown in the figure, the transit server includes a processor, a memory, a communication interface, and one or more programs. Or multiple programs are stored in the above-mentioned memory and are configured to be executed by the above-mentioned processor, and the above-mentioned programs include instructions for executing the following steps:

In a possible example, in the aspect of reading the search request message from the message queue of the storage server, the above program includes instructions for executing the following steps:

In a possible example, the above program also includes instructions for performing the following steps:

Obtain the health status data of all elastic search servers in the at least one elastic search server cluster, where the health status data includes at least one of the following health status sub-data: indexing rate, indexing time, memory size, number of request rejections, where: Each of the health status sub-data corresponds to a parameter threshold;

It is determined whether there is a health state sub-data in the health state data that exceeds a corresponding parameter threshold, and if it does not exist, the operation of reading the search request message from the message queue of the storage server is performed.

In a possible example, the search request data includes elastic search server cluster information, and in terms of sending the search request message to the first elastic search server in the at least one elastic search server cluster, the above program includes Instructions to perform the following steps:

Determining the first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

Sending the search request message to the first elastic search server.

In a possible example, the search request data includes elastic search server cluster information and elastic search engine version information, and in the sending of the search request message to the first elastic search server cluster in the at least one elastic search server cluster In terms of the search server, the above program includes instructions for performing the following steps:

Determine the elastic search server corresponding to the version information in the first elastic search server cluster;

The elastic search server corresponding to the version information is used as the first elastic search server, and the search request message is sent to the first elastic search server.

In a possible example, the transit server includes an SDK interface or an API interface, and in terms of sending the search request message to the first elastic search server, the above program includes instructions for executing the following steps:

Sending the search request message to the first elastic search server through the SDK interface; or,

The search request message is sent to the first elastic search server through the API interface.

In a possible example, if the transit server receives a feedback message that the first elastic search server cluster is out of the network abnormality, the above program further includes instructions for executing the following steps:

The foregoing mainly introduces the solution of the embodiment of the present application from the perspective of the execution process on the method side. It can be understood that, in order to implement the above-mentioned functions, an electronic device includes hardware structures and/or software modules corresponding to each function. Those skilled in the art should easily realize that in combination with the units and algorithm steps of the examples described in the embodiments provided herein, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

The embodiment of the present application may divide the electronic device into functional units according to the foregoing method examples. For example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The above-mentioned integrated unit can be implemented in the form of hardware or software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.

Please refer to FIG. 5. FIG. 5 is a schematic structural diagram of a data transmission device disclosed in an embodiment of the present application, which is applied to a storage server in a data transmission system. An elastic search server cluster. The data transmission system also includes a terminal device, a gateway server, a transit server, and at least one elastic search server cluster. The data transmission device includes a receiving unit 501, a writing unit 502, and a first sending unit 503. ,

The receiving unit 501 is configured to receive a search request message sent by the gateway server, where the search request message carries search request data, and the search request message is used by the terminal device to send the search request data to the gateway. Server, and then a message generated by the gateway server according to the search request data;

The writing unit 502 is configured to write the search request message into a message queue;

The receiving unit 501 is further configured to receive a subscription request sent by the transit server;

The first sending unit 503 is configured to send the search request message to the transit server according to the subscription request, and instruct the transit server to send the search request message to the at least one elastic search server cluster The first elastic search server in, the first elastic search server is any server in the at least one elastic search server cluster.

Optionally, the search request data includes elastic search server cluster information and index name information to which the search request data belongs. In the aspect of writing the search request message to the message queue, the writing unit 502 specifically uses At:

Writing the message set into the message queue;

It can be seen that the data transmission device described in the embodiment of this application sends a subscription request to the storage server; and reads the search request message from the message queue of the storage server. The search request message carries the search request data, and the search request message is The terminal device sends the search request data to the gateway server, and then the gateway server generates a search request message according to the search request data and writes it to the message queue of the storage server; the search request message is sent to the first in at least one elastic search server cluster An elastic search server. The first elastic search server is any server in at least one elastic search server cluster. In this way, the elastic search server can be improved by storing the search request data in the message queue and reading it by the relay server. The availability of the cluster prevents the entire elastic search server cluster from being unstable or unavailable due to the large amount of data written in the search request data. In addition, since the search request data is stored in the message queue, it is possible to backtrack the message for each search request data Track, when the elastic search server cluster is abnormal, you can also read the search request data from the message queue again to recover from the failure.

It should be noted that the storage server described in the embodiment of the present application is presented in the form of a functional unit. The term "unit" used here should be understood as the broadest possible meaning. The object used to realize the functions described by each "unit" may be, for example, an integrated circuit ASIC, a single circuit, used to execute one or more software or firmware. Program processor (shared, dedicated or chipset) and memory, combinatorial logic circuit, and/or provide other suitable components for realizing the above-mentioned functions.

Among them, the receiving unit 501, the writing unit 502, and the first sending unit 503 may be a control circuit or a processor.

Please refer to FIG. 6A. FIG. 6A is a schematic structural diagram of a data transmission device disclosed in an embodiment of the present application, which is applied to a transit server in a data transmission system. An elastic search server cluster, the data transmission device includes a second sending unit 601 and a reading unit 602, wherein:

The second sending unit 601 is configured to send a subscription request to the storage server;

The reading unit 602 is configured to read a search request message from a message queue of the storage server, the search request message carries search request data, and the search request message is sent by the terminal device. Data to the gateway server, and then the gateway server generates the search request message according to the search request data and writes it into the message queue of the storage server;

The second sending unit 602 is further configured to send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is the at least one elastic search server Any server in the cluster.

Optionally, in the reading the search request message surface from the message queue of the storage server, the reading unit is specifically configured to:

Optionally, as shown in FIG. 6B, FIG. 6B is a modified structure of the data transmission device shown in FIG. 6A. Compared with FIG. 6A, it may further include: an acquiring unit 603 and a determining unit 604, where

The acquiring unit 603 is configured to acquire health status data of all elastic search servers in the at least one elastic search server cluster, where the health status data includes at least one of the following health status sub-data: indexing rate, indexing time, memory Size and the number of request rejections, where each of the health status sub-data corresponds to a parameter threshold;

The determining unit 604 is configured to determine whether there is a health status sub-data that exceeds a corresponding parameter threshold in the health status data, if not, the reading unit 602 executes the message queue from the storage server Read the search request message in the operation.

Optionally, the search request data includes elastic search server cluster information, and in the aspect of sending the search request message to the first elastic search server in the at least one elastic search server cluster, the second sending unit Specifically used for:

Sending the search request message to the first elastic search server.

Optionally, the search request data includes elastic search server cluster information and version information of an elastic search engine, in the aspect of sending the search request message to the first elastic search server in the at least one elastic search server cluster , The second sending unit 601 is specifically configured to:

Optionally, the transit server includes an SDK interface or an API interface, and in terms of sending the search request message to the first elastic search server, the second sending unit 601 is specifically configured to:

Optionally, if the transit server receives a feedback message that the first elastic search server cluster is out of the network abnormality, the second sending unit 601 is further configured to:

It should be noted that the transit server described in the embodiment of the present application is presented in the form of a functional unit. The term "unit" used here should be understood as the broadest possible meaning. The object used to realize the functions described by each "unit" may be, for example, an integrated circuit ASIC, a single circuit, used to execute one or more software or firmware. Program processor (shared, dedicated or chipset) and memory, combinatorial logic circuit, and/or provide other suitable components for realizing the above-mentioned functions.

Wherein, the second sending unit 601, the reading unit 602, the acquiring unit 603, and the determining unit 603 may be a control circuit or a processor.

An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program causes a computer to execute a part of any data transmission method as described in the above method embodiment. Or all steps.

The embodiments of the present application also provide a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to cause a computer to execute the method described in the foregoing method embodiment. Part or all of the steps of any data transmission method.

It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited by the described sequence of actions. Because according to this application, some steps can be performed in other order or at the same time. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in an embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed device may be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or may be Integrate into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software program modules.

If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer readable memory. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory. A number of instructions are included to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The foregoing memory includes: U disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable memory, and the memory can include: a flash disk , ROM, RAM, magnetic disk or CD, etc.

The embodiments of the application are described in detail above, and specific examples are used in this article to illustrate the principles and implementation of the application. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the application; at the same time, for Those of ordinary skill in the art, based on the idea of the application, will have changes in the specific implementation and the scope of application. In summary, the content of this specification should not be construed as a limitation to the application.

Claims

A data transmission method, characterized in that it is applied to a storage server in a data transmission system, and the data transmission system further includes a terminal device, a gateway server, a transit server, and at least one elastic search server cluster, and the method includes:

Receiving a search request message sent by the gateway server, the search request message carrying search request data, the search request message is sent by the terminal device to the gateway server, and then the gateway server A message generated according to the search request data;

Writing the search request message into a message queue;

Receive a subscription request sent by the transit server, send the search request message to the transit server according to the subscription request, and instruct the transit server to send the search request message to the at least one elastic search server cluster The first elastic search server in, the first elastic search server is any server in the at least one elastic search server cluster.
The method according to claim 1, wherein the search request data includes elastic search server cluster information and index name information to which the search request data belongs, and writing the search request message to a message queue includes:

Dividing the search request message into message sets according to the elastic search server cluster information and the index name information;

Writing the message set into the message queue;

A mapping relationship between the name of the message set corresponding to the search request message and the name of the elastic search server cluster and the index is established.
A data transmission method, characterized in that it is applied to a transit server in a data transmission system, the data transmission system further includes a terminal device, a gateway server, a storage server, and at least one elastic search server cluster, and the method includes:

Send a subscription request to the storage server; and read a search request message from the message queue of the storage server, the search request message carries search request data, and the search request message is sent by the terminal device to the search Request data to the gateway server, and then the gateway server generates the search request message according to the search request data and writes it to the message queue of the storage server;

The search request message is sent to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.
The method according to claim 3, wherein said reading the search request message from the message queue of the storage server comprises:

According to the sequence of the message sets in the message queue, the message set is read from the message queue of the storage server, and the message set includes the search request message.
The method according to claim 3 or 4, wherein the method further comprises:

Obtain the health status data of all elastic search servers in the at least one elastic search server cluster, where the health status data includes at least one of the following health status sub-data: indexing rate, indexing time, memory size, number of request rejections, where: Each of the health status sub-data corresponds to a parameter threshold;

It is determined whether there is a health state sub-data in the health state data that exceeds a corresponding parameter threshold, and if it does not exist, the operation of reading the search request message from the message queue of the storage server is performed.
The method according to claim 5, wherein the search request data includes elastic search server cluster information, and the sending the search request message to the first elastic search server in the at least one elastic search server cluster, include:

Determining the first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

Sending the search request message to the first elastic search server.
The method according to claim 5, wherein the search request data includes elastic search server cluster information and elastic search engine version information, and the search request message is sent to the at least one elastic search server cluster The first elastic search server, including:

Determining the first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

Determine the elastic search server corresponding to the version information in the first elastic search server cluster;

The elastic search server corresponding to the version information is used as the first elastic search server, and the search request message is sent to the first elastic search server.
The method according to claim 6 or 7, wherein the transit server includes an SDK interface or an API interface, and the sending the search request message to the first elastic search server includes:

Sending the search request message to the first elastic search server through the SDK interface; or,

The search request message is sent to the first elastic search server through the API interface.
The method according to claim 6 or 7, wherein if the transit server receives a feedback message that the first elastic search server cluster is out of the network abnormality, the method further comprises:

The search request message is sent to a second elastic search server in a second elastic search server cluster in the at least one elastic search server cluster, where the second elastic search server cluster and the first elastic search server cluster are located Different computer rooms.
A data transmission device, characterized in that it is applied to a storage server in a data transmission system, the data transmission system further includes a terminal device, a gateway server, a transit server, and at least one elastic search server cluster, and the data transmission device includes:

The receiving unit is configured to receive a search request message sent by the gateway server, the search request message carries search request data, and the search request message is the terminal device sending the search request data to the gateway server, and then A message generated by the gateway server according to the search request data;

A writing unit for writing the search request message into the message queue;

The receiving unit is further configured to receive a subscription request sent by the transit server;

The first sending unit is configured to send the search request message to the transit server according to the subscription request, and instruct the transit server to send the search request message to the first in the at least one elastic search server cluster An elastic search server, and the first elastic search server is any server in the at least one elastic search server cluster.
The data transmission device according to claim 10, wherein the search request data includes elastic search server cluster information and index name information to which the search request data belongs, and the search request message is written in the message In terms of queues, the writing unit is specifically used for:

Dividing the search request message into message sets according to the elastic search server cluster information and the index name information;

Writing the message set into the message queue;

A mapping relationship between the name of the message set corresponding to the search request message and the name of the elastic search server cluster and the index is established.
A data transmission device, characterized in that it is applied to a transit server in a data transmission system, the data transmission system further includes a terminal device, a gateway server, a storage server, and at least one elastic search server cluster, and the data transmission device includes:

The second sending unit is configured to send a subscription request to the storage server;

The reading unit is configured to read a search request message from the message queue of the storage server, the search request message carries search request data, and the search request message is sent by the terminal device to the search request data. The gateway server, then the gateway server generates the search request message according to the search request data and writes it into the message queue of the storage server;

The second sending unit is further configured to send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is the at least one elastic search server cluster Any server in.
The data transmission device according to claim 12, wherein, in the reading of the search request message surface from the message queue of the storage server, the reading unit is specifically configured to:

According to the sequence of the message sets in the message queue, the message set is read from the message queue of the storage server, and the message set includes the search request message.
The data transmission device according to claim 11, wherein the device further comprises an acquiring unit and a determining unit, wherein:

The acquiring unit is configured to acquire health status data of all elastic search servers in the at least one elastic search server cluster, where the health status data includes at least one of the following health status sub-data: index rate, index time consumption, memory size The number of request rejections, wherein each of the health status sub-data corresponds to a parameter threshold;

The determining unit is configured to determine whether there are health status sub-data that exceeds a corresponding parameter threshold in the health status data, and if not, the reading unit executes the reading from the message queue of the storage server The operation of fetching the search request message.
The data transmission device according to claim 11, wherein the search request data includes elastic search server cluster information, and in the sending of the search request message to the first one in the at least one elastic search server cluster Regarding the elastic search server, the second sending unit is specifically configured to:

Determining the first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

Sending the search request message to the first elastic search server.
The data transmission device according to claim 11, wherein the search request data includes elastic search server cluster information and version information of the elastic search engine, and the search request message is sent to the at least one elastic search engine. Regarding the first elastic search server in the search server cluster, the second sending unit is specifically configured to:

Determining the first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

Determine the elastic search server corresponding to the version information in the first elastic search server cluster;

The elastic search server corresponding to the version information is used as the first elastic search server, and the search request message is sent to the first elastic search server.
A storage server, characterized by comprising a processor, a memory, a communication interface, and one or more programs, the one or more programs are stored in the memory and configured to be executed by the processor, The program includes instructions for performing the steps in the method as claimed in claim 1 or 2.
A transit server, characterized by comprising a processor, a memory, a communication interface, and one or more programs, the one or more programs are stored in the memory and configured to be executed by the processor, The program includes instructions for performing the steps in the method according to any one of claims 3-9.
A computer-readable storage medium, characterized by storing a computer program for electronic data exchange, wherein the computer program causes a computer to execute the method according to any one of claims 1-9.
A computer program product, wherein the computer program product comprises a non-transitory computer-readable storage medium storing a computer program, and the computer program causes a computer to execute the method according to any one of claims 1-9.