CN115087968A

CN115087968A - Data transmission method and device

Info

Publication number: CN115087968A
Application number: CN202080096690.9A
Authority: CN
Inventors: 郭子亮
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd; Shenzhen Huantai Technology Co Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd; Shenzhen Huantai Technology Co Ltd
Priority date: 2020-03-24
Filing date: 2020-03-24
Publication date: 2022-09-20
Also published as: WO2021189253A1

Abstract

The embodiment of the application discloses a data transmission method and a device, which are applied to a storage server in a data transmission system, wherein the data transmission system also comprises a terminal device, a gateway server, a transit server and at least one elastic search server cluster.

Description

Data transmission method and device

Technical Field

The present application relates to the field of electronic technologies, and in particular, to a data transmission method and apparatus.

Background

Currently, a terminal device of a user directly inserts or updates search request data into an index of an elastic search server using a presentation-layer state transition application program interface (restful api) provided by an elastic search (Es) engine or a client software development kit (client SDK interface). When the search request data is written into the elastic search server, a put interface is used for submitting a plurality of document data at one time, and a bulk interface is used for submitting one document data at one time.

When the flexible search server is accessed in the existing mode, the submitting speed and the data volume of the search request data are completely dependent on the frequency of calling the interface by the terminal equipment and the total number of data incoming in each request (using the bulk interface). Under the condition that the available resources of the elastic search server cluster are less, if the writing speed of the terminal device of the user is too high, the elastic search server cluster is likely to be unstable, and even the elastic search server cluster is not available, so that the usability of data writing or search service provided by using the elastic search server cluster is reduced, and the requirement of the elastic search server in the elastic search server cluster on the available time length is difficult to guarantee.

Disclosure of Invention

The embodiment of the application provides a data transmission method and device, which can improve the availability of an elastic search server cluster and avoid the phenomenon that the whole elastic search server cluster is unstable or unavailable due to the large data volume written in by search request data.

In a first aspect, an embodiment of the present application provides a data transmission method, which is applied to a storage server in a data transmission system, where the data transmission system further includes a terminal device, a gateway server, a relay server, and at least one elastic search server cluster, and the method includes:

receiving a search request message sent by the gateway server, wherein the search request message carries search request data, and the search request message is a message generated by the gateway server according to the search request data after the terminal device sends the search request data to the gateway server;

writing the search request message into a message queue;

receiving a subscription request sent by the transit server, sending the search request message to the transit server according to the subscription request, and instructing the transit server to send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.

In a second aspect, an embodiment of the present application provides a data transmission method, which is applied to a relay server in a data transmission system, where the data transmission system further includes a terminal device, a gateway server, a storage server, and at least one elastic search server cluster, and the method includes:

sending a subscription request to the storage server; reading a search request message from a message queue of the storage server, wherein the search request message carries search request data, and the search request message is sent to the gateway server by the terminal device, and then written into the message queue of the storage server after the gateway server generates the search request message according to the search request data;

sending the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.

In a third aspect, an embodiment of the present application provides a data transmission apparatus, which is applied to a storage server in a data transmission system, where the data transmission system further includes a terminal device, a gateway server, a relay server, and at least one elastic search server cluster, and the data transmission apparatus includes:

a receiving unit, configured to receive a search request message sent by the gateway server, where the search request message carries search request data, and the search request message is a message that is generated by the gateway server according to the search request data after the terminal device sends the search request data to the gateway server;

a writing unit, configured to write the search request message into a message queue;

the receiving unit is further configured to receive a subscription request sent by the transit server;

a first sending unit, configured to send the search request message to the relay server according to the subscription request, and instruct the relay server to send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.

In a fourth aspect, an embodiment of the present application provides a data transmission device, which is applied to a relay server in a data transmission system, where the data transmission system further includes a terminal device, a gateway server, a storage server, and at least one elastic search server cluster, and the data transmission device includes:

a second sending unit, configured to send a subscription request to the storage server;

a reading unit, configured to read a search request message from a message queue of the storage server, where the search request message carries search request data, and the search request message is obtained by sending, by the terminal device, the search request data to the gateway server, and then writing, by the gateway server, the search request message into the message queue of the storage server after generating, according to the search request data, the search request message;

the second sending unit is further configured to send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.

In a fifth aspect, an embodiment of the present application provides a storage server, including a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps in the first aspect of the embodiment of the present application.

In a sixth aspect, an embodiment of the present application provides a transit server, including a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps in the second aspect of the embodiment of the present application.

In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program for electronic data exchange, where the computer program enables a computer to perform some or all of the steps described in the first aspect or the second aspect of the embodiment of the present application.

In an eighth aspect, embodiments of the present application provide a computer program product, where the computer program product includes a non-transitory computer-readable storage medium storing a computer program, where the computer program is operable to cause a computer to perform some or all of the steps as described in the first or second aspects of embodiments of the present application. The computer program product may be a software installation package.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1A is a schematic architecture diagram of a data transmission system for implementing a data transmission method according to an embodiment of the present application;

fig. 1B is a schematic flowchart of a data transmission method disclosed in an embodiment of the present application;

fig. 1C is a schematic illustration showing a message queue written with the search request message according to an embodiment of the present application;

fig. 1D is a schematic diagram illustrating a mapping relationship between a message set name and an elastic search server cluster and an index name provided in an embodiment of the present application;

fig. 2 is a schematic flow chart of another data transmission method disclosed in the embodiments of the present application;

fig. 3 is a schematic flowchart of another data transmission method disclosed in the embodiment of the present application;

FIG. 4A is a schematic structural diagram of another storage server disclosed in the embodiments of the present application;

fig. 4B is a schematic structural diagram of another transit server disclosed in the embodiment of the present application;

fig. 5 is a schematic structural diagram of a data transmission device disclosed in an embodiment of the present application;

fig. 6A is a schematic structural diagram of another data transmission device disclosed in the embodiment of the present application;

fig. 6B is a modified structure of the data transmission device shown in fig. 6A disclosed in the embodiment of the present application.

Detailed Description

In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The terms "first," "second," and the like in the description and claims of the present application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

In order to facilitate better understanding of the technical solutions described in the present invention, the following explains technical terms related to the embodiments of the present invention:

elastic search (Es) server: the Es server is a search server based on a Lucene engine, provides a distributed full-text search engine with multi-user capability, and is a search engine framework supporting high availability and expandability.

Message Queue (MQ): is a "first-in-first-out" data mechanism in the underlying data structure.

Presentation layer state transformation application program interface (presentation state transfer-full application programming interface, restfull api): the presentation layer state transition application program interface has a software build style designed to provide web services.

Software Development Kit (SDK): typically a collection of development tools used by software engineers to create application software for a particular software package, software framework, hardware platform, operating system, etc.

The terminal device related to the embodiment of the present application may include various handheld devices (such as a Mobile phone, a tablet computer, etc.), vehicle-mounted devices, wearable devices, computing devices, or other processing devices connected to a wireless modem, and various forms of User Equipment (UE), a Mobile Station (MS), and the like.

The following describes embodiments of the present application in detail.

The data transmission method provided by the embodiment of the application is suitable for a scene that the terminal equipment of the user accesses the elastic search server through the search request data. Fig. 1A is a schematic structural diagram of a data transmission system for implementing a data transmission method according to an embodiment of the present invention. The data transmission system may include a terminal device 1001, a gateway server 1002, a storage server 1003, a transit server 1004, and at least one flexible search server cluster, each of the at least one flexible search server cluster including a plurality of flexible search servers 1004, wherein the terminal device 1001 is a client device used by a user, the gateway server 1002 is connected to the terminal device 1001, the terminal device 1001 does not directly transmit search request data to the flexible search server cluster but transmits the search request data to the gateway server 1002, the storage server 1003 is connected to the gateway server 1002, the gateway server 1002 writes the received search request data in the storage server 1003 in the form of a search request message, the storage server 1003 writes the search request message in a message queue, the storage server 1003 is further connected to the transit server 1004, the relay server 1004 enables the terminal device of the user to access the elastic search server cluster by sending a subscription message to the storage server 1003, reading a search request message from the storage server 1003, and then sending the read search request message to the elastic search server 1005 in the elastic search server cluster by the relay server 1004.

Referring to fig. 1B, fig. 1B is a schematic flowchart of a data transmission method disclosed in an embodiment of the present application, and is applied to a storage server in a data transmission system, where the data transmission system further includes a terminal device, a gateway server, a relay server, and at least one flexible search server cluster, and the data transmission method includes the following steps:

101. receiving a search request message sent by the gateway server, wherein the search request message carries search request data, and the search request message is a message generated by the gateway server according to the search request data after the terminal device sends the search request data to the gateway server.

In the embodiment of the application, when a user accesses the elastic search server cluster through the terminal device, the user service of the terminal device may send search request data to the gateway server, and then the gateway server generates a search request message according to the search request data and sends the search request message to the storage server, so that the storage server may receive the search request message sent by the gateway server.

The terminal device can register user service in advance, and when the user service is registered, the terminal device can acquire the elastic search server cluster information corresponding to the index in advance, so that the terminal device can send a search request message carrying the elastic search server cluster information to the gateway server when the terminal device needs to perform search access.

102. And writing the search request message into a message queue.

The storage server can receive different search request messages sent by the gateway server, so that the storage server can write the search request messages into the message queue in sequence according to the sequence of receiving the search request messages.

Optionally, the search request data includes flexible search server cluster information and index name information to which the search request data belongs, and in step 102, the writing the search request message into the message queue may include the following steps:

11. dividing the search request message into a message set according to the elastic search server cluster information and the index name information;

12. writing the set of messages to the message queue;

13. and establishing a mapping relation between the message set name corresponding to the search request message and the cluster and index name of the elastic search server.

Please refer to fig. 1C for a schematic illustration of a demonstration of writing the search request message into a message queue according to rules 91 and 25.05.2020, where the storage server divides the search request message and other search request messages with the same index name and elastic search server cluster information received by the storage server into the same message set according to the index name and the elastic search server cluster information, so that one message set in the message queue corresponds to one message set name, and all the search request messages in each message set have the same index name and the elastic search server cluster information, and the storage server further establishes a mapping relationship between the message set name and the elastic search server cluster and index name corresponding to the search request message, as shown in fig. 1D, fig. 1D is a schematic diagram illustrating a mapping relationship between a message set name and an elastic search server cluster and an index name provided in an embodiment of the present application, where a topic a, a topic B, a topic C, and a topic D are message set names, C1, C2, C3. are elastic search server clusters, and V1, V2, and V3. are index names, and by establishing the mapping relationship, a relay server can read a message set name, an elastic search server cluster, and an index name corresponding to a message set to which a search request message belongs at the same time when reading the search request message in a message queue.

103. Receiving a subscription request sent by the transit server, sending the search request message to the transit server according to the subscription request, and instructing the transit server to send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.

The transit server may be, for example, a sink server.

The storage server may receive a subscription request sent by the transit server, where the subscription request is used to instruct the storage server to send a search request message in the message queue to the transit server.

In a specific implementation, the storage server can read the search request sets according to the arrangement sequence of the message sets in the message queue, and the sequence of the search request data read from the message queue and the sequence of the search request data written into the message queue can be ensured to be consistent as the read data and the write data in the message queue follow the rule of first-in first-out.

Considering that different users may access the resilient search server cluster through different terminal devices, in the case of a large amount of search request data, a part of the search request data may be stored through the message queue, and the message queue may store, for the load condition of the current flexible search server cluster, the search request data is stored in the message queue, read by the transit server, the availability of the elastic search server cluster can be improved, the phenomenon that the whole elastic search server cluster is unstable or unavailable due to the large data volume written by the search request data can be avoided, and in addition, since the search request data is stored in the message queue, the trajectory of the message can be traced back for each piece of search request data, when the elastic search server cluster is abnormal, the search request data can be read again from the message queue, so that fault recovery is performed.

It can be seen that the data transmission method described in the embodiments of the present application is applied to a storage server in a data transmission system, where the data transmission system further includes a terminal device, a gateway server, a relay server, and at least one elastic search server cluster, and by receiving a search request message sent by the gateway server, the search request message carries search request data, where the search request message is a message generated by the gateway server according to the search request data after the terminal device sends the search request data to the gateway server; writing the search request message into a message queue; receiving a subscription request sent by a transit server, sending a search request message to the transit server according to the subscription request, and instructing the transit server to send the search request message to a first elastic search server in at least one elastic search server cluster, wherein the first elastic search server is any server in the at least one elastic search server cluster, so that the availability of the elastic search server cluster can be improved by storing search request data in a message queue and reading the search request data by the transit server, the situation that the whole elastic search server cluster is unstable or unavailable due to large data volume written by the search request data is avoided, in addition, the track of the message can be traced back for each piece of search request data due to the fact that the search request data is stored in the message queue, and when the elastic search server cluster is abnormal, the search request data can be read again from the message queue, thereby performing failure recovery.

Consistent with the above, fig. 2 is a schematic flow chart of a data transmission method disclosed in the embodiment of the present application. The data transmission method is applied to a transfer server in a data transmission system, the data transmission system also comprises terminal equipment, a gateway server, a storage server and at least one elastic search server cluster, and the data transmission method comprises the following steps:

201. sending a subscription request to the storage server; and reading a search request message from a message queue of the storage server, wherein the search request message carries search request data, and the search request message is obtained by sending the search request data to the gateway server by the terminal device, and then writing the search request message into the message queue of the storage server by the gateway server after generating the search request message according to the search request data.

In specific implementation, when a user accesses the elastic search server cluster through the terminal device, the user service of the terminal device may send search request data to the gateway server, then the gateway server generates a search request message according to the search request data and sends the search request message to the storage server, and further, the server may write the search request message into a message queue, so that the transit server may send a subscription request to the storage server; and reads the search request message from the message queue of the storage server.

Optionally, in step 201, the reading the search request message from the message queue of the storage server may include the following steps:

and reading the message set from the message queue of the storage server according to the arrangement sequence of the message set in the message queue, wherein the message set comprises the search request message.

The search request data comprises elastic search server cluster information and index name information to which the search request data belongs, and when a storage server writes a search request message into a message queue, the storage server divides the search request message into message sets according to the elastic search server cluster information and the index name information, and then writes the message sets into the message queue, so that the relay server can read the message sets from the message queue of the storage server according to the arrangement sequence of the message sets in the message queue.

Optionally, in step 201, before reading the search request message from the message queue of the storage server, the method may further include the following steps:

2011. acquiring health state data of all elastic search servers in the at least one elastic search server cluster, wherein the health state data comprises at least one of the following health state sub-data: the method comprises the steps of indexing speed, indexing time consumption, memory size and request rejection number, wherein each piece of health state subdata corresponds to a parameter threshold;

2012. and determining whether the health state data contains health state sub-data exceeding a corresponding parameter threshold, and if not, executing the operation of reading the search request message from the message queue of the storage server.

The transit server can obtain the current index rate, index time consumption, memory size and request rejection number of all the elastic search servers in at least one elastic search server cluster, wherein the index rate refers to the average value of a plurality of index rates corresponding to all the elastic search servers, the index time consumption refers to the sum of a plurality of index time consumptions corresponding to all the elastic search servers, the memory size refers to the sum of a plurality of memory sizes corresponding to all the elastic search servers, and the request rejection number refers to the sum of a plurality of request rejection numbers corresponding to all the elastic search servers. The transit server may also preset a first parameter threshold corresponding to the index rate, a second parameter threshold corresponding to the index consumption time, a third parameter threshold corresponding to the memory size, and a fourth parameter threshold corresponding to the request rejection number. And then respectively determining whether the index rate, the index consumed time, the memory size and the request rejected number exceed corresponding parameter thresholds, if the index rate, the index consumed time, the memory size and the request rejected number do not exceed the corresponding parameter thresholds, reading the search request message from the message queue of the storage server, and if at least one health state subdata exceeds the corresponding parameter threshold, suspending the execution of the operation of reading the search request message from the message queue of the storage server, so that the reading rate of the search request message can be dynamically controlled, and the situation that the elastic search server cluster is unavailable due to the fact that the writing request amount is too large is avoided.

202. And sending the search request message to a first elastic search server in the at least one elastic search server cluster, wherein the first elastic search server is any server in the at least one elastic search server cluster.

The transit server may send the read search request message to a first flexible search server in the at least one flexible search server cluster, where the first flexible search server may be any server in the at least one flexible search server cluster.

Optionally, the search request data includes elastic search server cluster information, and the sending the search request message to the first elastic search server in the at least one elastic search server cluster in step 202 may include the following steps:

21. determining a first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

22. sending the search request message to the first flexible search server.

The search request data may include information of an elastic search server cluster to be sent by the search request data, so that the dedicated server can determine a first elastic search server cluster in the at least one elastic search server cluster according to the information of the elastic search server cluster, where the first elastic search server cluster is an elastic search server cluster corresponding to the information of the elastic search server cluster. The relay server may then send the search request message to a first elastic search server of a first cluster of elastic search servers, which may be any server of the first cluster of elastic search servers.

Optionally, the step 202 of sending the search request message to the first flexible search server in the at least one flexible search server cluster may include the following steps:

23. determining a first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

24. determining an elastic search server corresponding to the version information in the first elastic search server cluster;

25. and taking the elastic search server corresponding to the version information as the first elastic search server, and sending the search request message to the first elastic search server.

The gateway server can support a plurality of elastic search engine versions, the search request data can include version information of the elastic search engines, and then when the search request data is sent to the gateway server from the terminal device and then reaches the transfer server through the storage server, the transfer server can determine a first elastic search server corresponding to the version information in the first elastic search server cluster according to the version information of the elastic search engines in the search request data. Therefore, the user service of the terminal equipment can operate the elastic search server clusters with different versions through the same interface, and the cost can be reduced in the cluster upgrading and version switching processes in the later period.

Optionally, the transit server includes an SDK interface or an API interface, and the sending the search request message to the first flexible search server in step 22 or step 25 may include the following steps:

a1, sending the search request message to the first elastic search server through the SDK interface; or,

a2, sending the search request message to the first elastic search server through the API interface.

The relay server may include an SDK interface or an API interface, so that the relay server may send the search request message to the first flexible search server through the SDK interface or send the search request message to the first flexible search server through the API interface.

Optionally, if the transit server receives a feedback message that the first elastic search server cluster is abnormal in the outgoing network, in this embodiment of the present application, the method may further include the following steps:

and sending the search request message to a second elastic search server in a second elastic search server cluster in the at least one elastic search server cluster, wherein the second elastic search server cluster and the first elastic search server cluster are in different machine rooms.

The transit server can write the same search request message into different elastic search server clusters, and if the first elastic search server cluster is abnormal in an outgoing network, the transit server can also send the search request message to a second elastic search server in a second elastic search server cluster, and the second elastic search server can be any server in the second elastic search server cluster, so that the search request data sent by a user through the terminal device can be successfully accessed, and the situation that the elastic search server cluster is unavailable is reduced to the maximum extent.

It can be seen that the data transmission method described in the embodiments of the present application is applied to a relay server in a data transmission system, where the data transmission system further includes a terminal device, a gateway server, a storage server, and at least one elastic search server cluster, and sends a subscription request to the storage server; reading a search request message from a message queue of the storage server, wherein the search request message carries search request data, the search request message is obtained by sending the search request data to the gateway server by the terminal equipment, and then writing the search request message into the message queue of the storage server by the gateway server after generating the search request message according to the search request data; the search request message is sent to a first elastic search server in at least one elastic search server cluster, and the first elastic search server is any server in the at least one elastic search server cluster, so that the availability of the elastic search server cluster can be improved in a mode that the search request data is stored in the message queue and read by the transfer server, the phenomenon that the whole elastic search server cluster is unstable or unavailable due to large data volume written by the search request data is avoided, in addition, the search request data is stored in the message queue, the track of the message can be traced back for each piece of search request data, and when the elastic search server cluster is abnormal, the search request data can be read from the message queue again, so that fault recovery is performed.

Consistent with the above, fig. 3 is a schematic flow chart of a data transmission method disclosed in the embodiment of the present application. The data transmission method is applied to the data transmission system shown in fig. 1A, and the data transmission system can be a terminal device, a gateway server, a storage server, a transit server and at least one elastic search server cluster, and the data transmission method comprises the following steps:

301. the terminal device sends the search request data to the gateway server.

302. And the gateway server generates a search request message according to the search request data.

303. And the gateway server sends the search request message to a storage server.

304. And the storage server writes the search request message into a message queue.

305. And the transit server sends a subscription request to the storage server.

306. And the transit server reads the search request message from the message queue of the storage server.

307. And sending the search request message to a first elastic search server in the at least one elastic search server cluster, wherein the first elastic search server is any server in the at least one elastic search server cluster.

The detailed descriptions of steps 301 to 307 may refer to the corresponding descriptions of the data transmission methods described in fig. 1B and fig. 2, and are not repeated herein.

It can be seen that, in the data transmission method described in this embodiment of the present application, search request data is sent to a gateway server by a terminal device, the gateway server generates a search request message according to the search request data, the gateway server sends the search request message to a storage server, the storage server writes the search request message into a message queue, a relay server sends a subscription request to the storage server, the relay server reads the search request message from the message queue of the storage server and sends the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster, so that the search request data can be stored in the message queue, in addition, because the search request data is stored in the message queue, the track of the message can be traced back for each piece of search request data, and when the elastic search server cluster is abnormal, the search request data can also be read from the message queue again, so that fault recovery is performed.

Referring to fig. 4A, fig. 4A is a schematic structural diagram of a storage server disclosed in an embodiment of the present application, and as shown in the drawing, the storage server includes a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for performing the following steps:

writing the search request message into a message queue;

In one possible example, the search request data includes flexible search server cluster information and index name information to which the search request data belongs, and the program is specifically configured to execute the following instructions in terms of writing the search request message to a message queue:

dividing the search request message into a message set according to the elastic search server cluster information and the index name information;

writing the set of messages to the message queue;

and establishing a mapping relation between the message set name corresponding to the search request message and the cluster and index name of the elastic search server.

Referring to fig. 4B, fig. 4B is a schematic structural diagram of a relay server according to an embodiment of the present disclosure, and as shown in the drawing, the relay server includes a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the following steps:

and sending the search request message to a first elastic search server in the at least one elastic search server cluster, wherein the first elastic search server is any server in the at least one elastic search server cluster.

In one possible example, in the reading of the search request message from the message queue of the storage server, the above program includes instructions for performing the steps of:

In one possible example, the program further includes instructions for performing the steps of:

acquiring health state data of all elastic search servers in the at least one elastic search server cluster, wherein the health state data comprises at least one of the following health state sub-data: the method comprises the steps of indexing speed, indexing time consumption, memory size and request rejection number, wherein each piece of health state subdata corresponds to a parameter threshold;

and determining whether the health state data contains the health state subdata exceeding the corresponding parameter threshold, and if not, executing the operation of reading the search request message from the message queue of the storage server.

In one possible example, the search request data includes elastic search server cluster information, and the program includes instructions for performing, in connection with the sending of the search request message to a first elastic search server of the at least one elastic search server cluster:

determining a first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

sending the search request message to the first flexible search server.

In one possible example, the search request data includes flexible search server cluster information and flexible search engine version information, and the program includes instructions for performing, in connection with the sending of the search request message to a first flexible search server of the at least one flexible search server cluster:

determining an elastic search server corresponding to the version information in the first elastic search server cluster;

and taking the elastic search server corresponding to the version information as the first elastic search server, and sending the search request message to the first elastic search server.

In one possible example, the transit server includes an SDK interface or an API interface, and the program includes instructions for performing the following steps in the sending of the search request message to the first flexible search server:

sending the search request message to the first elastic search server through the SDK interface; or,

and sending the search request message to the first elastic search server through the API.

In a possible example, if the transit server receives a feedback message of an offline network anomaly of the first elastic search server cluster, the program further includes instructions for performing the following steps:

The above description has introduced the solution of the embodiment of the present application mainly from the perspective of the method-side implementation process. It is understood that the electronic device comprises corresponding hardware structures and/or software modules for performing the respective functions in order to realize the above-mentioned functions. Those of skill in the art will readily appreciate that the present application is capable of hardware or a combination of hardware and computer software implementing the various illustrative elements and algorithm steps described in connection with the embodiments provided herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiment of the present application, the electronic device may be divided into the functional units according to the method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit. It should be noted that the division of the unit in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation.

Referring to fig. 5, fig. 5 is a schematic structural diagram of a data transmission apparatus disclosed in an embodiment of the present application, which is applied to a storage server in a data transmission system, where the data transmission system further includes a terminal device, a gateway server, a relay server, and at least one elastic search server cluster, the data transmission apparatus includes a receiving unit 501, a writing unit 502, and a first sending unit 503, where,

the receiving unit 501 is configured to receive a search request message sent by the gateway server, where the search request message carries search request data, and the search request message is a message that is generated by the gateway server according to the search request data and is sent by the terminal device to the gateway server;

the writing unit 502 is configured to write the search request message into a message queue;

the receiving unit 501 is further configured to receive a subscription request sent by the transit server;

the first sending unit 503 is configured to send the search request message to the relay server according to the subscription request, and instruct the relay server to send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.

Optionally, the search request data includes elastic search server cluster information and index name information to which the search request data belongs, and in terms of writing the search request message into a message queue, the writing unit 502 is specifically configured to:

writing the set of messages to the message queue;

It can be seen that, the data transmission apparatus described in the embodiment of the present application sends a subscription request to the storage server; reading a search request message from a message queue of the storage server, wherein the search request message carries search request data, the search request message is obtained by sending the search request data to the gateway server by the terminal equipment, and then writing the search request message into the message queue of the storage server by the gateway server after generating the search request message according to the search request data; the method comprises the steps of sending a search request message to a first elastic search server in at least one elastic search server cluster, wherein the first elastic search server is any server in the at least one elastic search server cluster, so that the availability of the elastic search server cluster can be improved in a mode that search request data are stored in a message queue and read by a transfer server, the phenomenon that the whole elastic search server cluster is unstable or unavailable due to the fact that the data volume written in the search request data is large is avoided, in addition, due to the fact that the search request data are stored in the message queue, the track of the message can be traced back for each piece of search request data, when the elastic search server cluster is abnormal, the search request data can be read from the message queue again, and therefore fault recovery is conducted.

It should be noted that the storage server described in the embodiments of the present application is presented in the form of a functional unit. The term "unit" as used herein is to be understood in its broadest possible sense, and objects used to implement the functions described by the respective "unit" may be, for example, an integrated circuit ASIC, a single circuit, a processor (shared, dedicated, or chipset) and memory that execute one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.

The receiving unit 501, the writing unit 502, and the first sending unit 503 may be a control circuit or a processor.

Referring to fig. 6A, fig. 6A is a schematic structural diagram of a data transmission apparatus, which is applied to a relay server in a data transmission system, where the data transmission system further includes a terminal device, a gateway server, a storage server, and at least one elastic search server cluster, the data transmission apparatus includes a second sending unit 601 and a reading unit 602, where,

the second sending unit 601 is configured to send a subscription request to the storage server;

the reading unit 602 is configured to read a search request message from a message queue of the storage server, where the search request message carries search request data, and the search request message is sent by the terminal device to the gateway server, and then is written into the message queue of the storage server after the gateway server generates the search request message according to the search request data;

the second sending unit 602 is further configured to send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.

Optionally, in the reading of the search request message surface from the message queue of the storage server, the reading unit is specifically configured to:

Alternatively, as shown in fig. 6B, fig. 6B is a modified structure of the data transmission device shown in fig. 6A, which may further include, compared with fig. 6A: an acquisition unit 603 and a determination unit 604, wherein,

the obtaining unit 603 is configured to obtain health status data of all elastic search servers in the at least one elastic search server cluster, where the health status data includes at least one of the following health status sub-data: the method comprises the steps of indexing speed, indexing time consumption, memory size and request rejection number, wherein each piece of health state subdata corresponds to a parameter threshold;

the determining unit 604 is configured to determine whether there is sub-health status data exceeding a corresponding parameter threshold in the health status data, and if there is no sub-health status data, the reading unit 602 executes the operation of reading the search request message from the message queue of the storage server.

Optionally, the search request data includes elastic search server cluster information, and in the aspect of sending the search request message to a first elastic search server in the at least one elastic search server cluster, the second sending unit is specifically configured to:

sending the search request message to the first flexible search server.

Optionally, the search request data includes elastic search server cluster information and version information of an elastic search engine, and in terms of sending the search request message to a first elastic search server in the at least one elastic search server cluster, the second sending unit 601 is specifically configured to:

Optionally, the transit server includes an SDK interface or an API interface, and in the aspect of sending the search request message to the first flexible search server, the second sending unit 601 is specifically configured to:

Optionally, if the relay server receives a feedback message that the first elastic search server cluster is abnormal in the outgoing network, the second sending unit 601 is further configured to:

It should be noted that the transit server described in the embodiments of the present application is presented in the form of a functional unit. The term "unit" as used herein should be understood in its broadest possible sense, and objects used to implement the functionality described in each "unit" may be, for example, an integrated circuit ASIC, a single circuit, a processor (shared, dedicated, or group) and memory that execute one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.

Among them, the second transmitting unit 601, the reading unit 602, the obtaining unit 603, and the determining unit 603 may be a control circuit or a processor.

Embodiments of the present application also provide a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any one of the data transmission methods as described in the above method embodiments.

Embodiments of the present application also provide a computer program product comprising a non-transitory computer readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the data transmission methods as recited in the above method embodiments.

It should be noted that for simplicity of description, the above-mentioned embodiments of the method are described as a series of acts, but those skilled in the art should understand that the present application is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implementing, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed coupling or direct coupling or communication connection between each other may be through some interfaces, indirect coupling or communication connection between devices or units, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may be implemented in the form of a software program module.

The integrated units, if implemented in the form of software program modules and sold or used as stand-alone products, may be stored in a computer readable memory. Based on such understanding, the technical solution of the present application may be substantially implemented or a part of or all or part of the technical solution contributing to the prior art may be embodied in the form of a software product stored in a memory, and including several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned memory comprises: various media capable of storing program codes, such as a usb disk, a read-only memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and the like.

Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable memory, which may include: flash disk, ROM, RAM, magnetic or optical disk, and the like.

The foregoing detailed description of the embodiments of the present application has been presented to illustrate the principles and implementations of the present application, and the above description of the embodiments is only provided to help understand the method and the core concept of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

A data transmission method is applied to a storage server in a data transmission system, the data transmission system further comprises a terminal device, a gateway server, a transit server and at least one elastic search server cluster, and the method comprises the following steps:

receiving a search request message sent by the gateway server, wherein the search request message carries search request data, and the search request message is a message generated by the gateway server according to the search request data after the terminal device sends the search request data to the gateway server;

writing the search request message into a message queue;

receiving a subscription request sent by the transit server, sending the search request message to the transit server according to the subscription request, and instructing the transit server to send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.
The method of claim 1, wherein the search request data includes flexible search server cluster information and index name information to which the search request data belongs, and the writing the search request message to a message queue includes:

dividing the search request message into a message set according to the elastic search server cluster information and the index name information;

writing the set of messages to the message queue;

and establishing a mapping relation between the message set name corresponding to the search request message and the cluster and index name of the elastic search server.
A data transmission method is characterized in that the method is applied to a transfer server in a data transmission system, the data transmission system further comprises a terminal device, a gateway server, a storage server and at least one elastic search server cluster, and the method comprises the following steps:

sending a subscription request to the storage server; reading a search request message from a message queue of the storage server, wherein the search request message carries search request data, and the search request message is sent to the gateway server by the terminal device, and then written into the message queue of the storage server after the gateway server generates the search request message according to the search request data;

and sending the search request message to a first elastic search server in the at least one elastic search server cluster, wherein the first elastic search server is any server in the at least one elastic search server cluster.
The method of claim 3, wherein reading the search request message from the message queue of the storage server comprises:

and reading the message set from the message queue of the storage server according to the arrangement sequence of the message set in the message queue, wherein the message set comprises the search request message.
The method of claim 3 or 4, further comprising:

acquiring health state data of all elastic search servers in the at least one elastic search server cluster, wherein the health state data comprises at least one of the following health state subdata: the method comprises the steps of indexing speed, indexing time consumption, memory size and request rejection number, wherein each piece of health state subdata corresponds to a parameter threshold;

and determining whether the health state data contains health state sub-data exceeding a corresponding parameter threshold, and if not, executing the operation of reading the search request message from the message queue of the storage server.
The method of claim 5, wherein the search request data includes elastic search server cluster information, and wherein sending the search request message to a first elastic search server of the at least one elastic search server cluster comprises:

determining a first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

sending the search request message to the first flexible search server.
The method of claim 5, wherein the search request data includes flexible search server cluster information and version information of a flexible search engine, and wherein sending the search request message to a first flexible search server of the at least one flexible search server cluster comprises:

determining a first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

determining an elastic search server corresponding to the version information in the first elastic search server cluster;

and taking the elastic search server corresponding to the version information as the first elastic search server, and sending the search request message to the first elastic search server.
The method of claim 6 or 7, wherein the transit server includes an SDK interface or an API interface, and wherein sending the search request message to the first flexible search server includes:

sending the search request message to the first elastic search server through the SDK interface; or,

and sending the search request message to the first elastic search server through the API.
The method according to claim 6 or 7, wherein if the transit server receives a feedback message of the first elastic search server cluster offline network anomaly, the method further comprises:

and sending the search request message to a second elastic search server in a second elastic search server cluster in the at least one elastic search server cluster, wherein the second elastic search server cluster and the first elastic search server cluster are in different machine rooms.
A data transmission device is characterized in that the data transmission device is applied to a storage server in a data transmission system, the data transmission system further comprises a terminal device, a gateway server, a transfer server and at least one elastic search server cluster, and the data transmission device comprises:

a receiving unit, configured to receive a search request message sent by the gateway server, where the search request message carries search request data, and the search request message is a message that is generated by the gateway server according to the search request data after the terminal device sends the search request data to the gateway server;

a writing unit, configured to write the search request message into a message queue;

the receiving unit is further configured to receive a subscription request sent by the transit server;

a first sending unit, configured to send the search request message to the relay server according to the subscription request, and instruct the relay server to send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.
The data transmission apparatus according to claim 10, wherein the search request data includes flexible search server cluster information and index name information to which the search request data belongs, and in terms of the writing of the search request message into a message queue, the writing unit is specifically configured to:

dividing the search request message into a message set according to the elastic search server cluster information and the index name information;

writing the set of messages to the message queue;

and establishing a mapping relation between the message set name corresponding to the search request message and the cluster and index name of the elastic search server.
A data transmission device is applied to a transfer server in a data transmission system, the data transmission system further comprises a terminal device, a gateway server, a storage server and at least one elastic search server cluster, and the data transmission device comprises:

a second sending unit, configured to send a subscription request to the storage server;

a reading unit, configured to read a search request message from a message queue of the storage server, where the search request message carries search request data, and the search request message is obtained by sending, by the terminal device, the search request data to the gateway server, and then writing, by the gateway server, the search request message into the message queue of the storage server after generating, according to the search request data, the search request message;

the second sending unit is further configured to send the search request message to a first elastic search server in the at least one elastic search server cluster, where the first elastic search server is any server in the at least one elastic search server cluster.
The data transmission apparatus according to claim 12, wherein, in the reading of the search request message surface from the message queue of the storage server, the reading unit is specifically configured to:

and reading the message set from the message queue of the storage server according to the arrangement sequence of the message set in the message queue, wherein the message set comprises the search request message.
The data transmission apparatus according to claim 11, wherein the apparatus further comprises an acquisition unit and a determination unit, wherein,

the acquiring unit is configured to acquire health status data of all elastic search servers in the at least one elastic search server cluster, where the health status data includes at least one of the following health status sub-data: the method comprises the steps of indexing speed, indexing time consumption, memory size and request rejection number, wherein each piece of health state subdata corresponds to a parameter threshold;

the determining unit is configured to determine whether there is sub-health status data exceeding a corresponding parameter threshold in the health status data, and if there is no sub-health status data, the reading unit executes the operation of reading the search request message from the message queue of the storage server.
The data transmission apparatus according to claim 11, wherein the search request data includes elastic search server cluster information, and in terms of the sending of the search request message to a first elastic search server in the at least one elastic search server cluster, the second sending unit is specifically configured to:

determining a first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

sending the search request message to the first flexible search server.
The data transmission apparatus according to claim 11, wherein the search request data includes flexible search server cluster information and version information of a flexible search engine, and in terms of the sending of the search request message to a first flexible search server in the at least one flexible search server cluster, the second sending unit is specifically configured to:

determining a first elastic search server cluster in the at least one elastic search server cluster according to the elastic search server cluster information;

determining an elastic search server corresponding to the version information in the first elastic search server cluster;

and taking the elastic search server corresponding to the version information as the first elastic search server, and sending the search request message to the first elastic search server.
A storage server comprising a processor, memory, a communications interface, and one or more programs stored in the memory and configured for execution by the processor, the programs comprising instructions for performing the steps in the method of claim 1 or 2.
A transit server comprising a processor, a memory, a communications interface, and one or more programs stored in the memory and configured for execution by the processor, the programs comprising instructions for performing the steps in the method of any of claims 3-9.
A computer-readable storage medium, characterized in that a computer program for electronic data exchange is stored, wherein the computer program causes a computer to perform the method according to any one of claims 1-9.
A computer program product, characterized in that the computer program product comprises a non-transitory computer readable storage medium storing a computer program, the computer program causing a computer to perform the method according to any one of claims 1-9.