CN113297110A - Data acquisition system, method and device - Google Patents

Data acquisition system, method and device Download PDF

Info

Publication number
CN113297110A
CN113297110A CN202110533607.6A CN202110533607A CN113297110A CN 113297110 A CN113297110 A CN 113297110A CN 202110533607 A CN202110533607 A CN 202110533607A CN 113297110 A CN113297110 A CN 113297110A
Authority
CN
China
Prior art keywords
data
index
storage module
data storage
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110533607.6A
Other languages
Chinese (zh)
Inventor
李靖轩
宋卓
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Innovation Co
Original Assignee
Alibaba Singapore Holdings Pte Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Singapore Holdings Pte Ltd filed Critical Alibaba Singapore Holdings Pte Ltd
Priority to CN202110533607.6A priority Critical patent/CN113297110A/en
Publication of CN113297110A publication Critical patent/CN113297110A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
    • G06F13/28—Handling requests for interconnection or transfer for access to input/output bus using burst mode transfer, e.g. direct memory access DMA, cycle steal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/602—Providing cryptographic facilities or services
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G06F9/5022—Mechanisms to release resources
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Bioethics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer And Data Communications (AREA)

Abstract

The embodiment of the specification provides a data acquisition system, a method and a device, wherein the data acquisition system comprises: an application server, a remote server; the application server is configured to establish a data transmission channel with the remote server, send registration information of a data storage module and identification information of the index to be detected to the remote server through the data transmission channel, collect index data of the index to be detected and store the index data in the data storage module; and the remote server is configured to call a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information, process the index data according to a preset data processing strategy and display a generated processing result.

Description

Data acquisition system, method and device
Technical Field
The embodiment of the specification relates to the technical field of computers, in particular to a data acquisition system. One or more embodiments of the present specification also relate to a data acquisition method, a data acquisition apparatus, a computing device, and a computer-readable storage medium.
Background
With the rapid development of information technology, a large number of devices are generally involved in the execution of a project in a network system in many fields (e.g., finance, communication, production, transportation, etc.). In order to ensure proper project execution, time consistency and accuracy between devices are generally required. In this case, it is necessary for each apparatus to continuously acquire time information from the server.
In the conventional TCP (Transmission Control Protocol)/IP technology, a large amount of server resources and memory bus bandwidth are occupied by an operating system and other software layers of an application server during data processing, data is copied and moved back and forth between a system memory of the application server and a cache of a remote server, heavy burden is imposed on a CPU and a memory of the application server, and especially mismatching between network bandwidth, processor speed and memory bandwidth may cause network communication delay.
Disclosure of Invention
In view of this, the present specification provides a data acquisition system. One or more embodiments of the present disclosure also relate to a data acquisition method, a data acquisition apparatus, a computing device, and a computer-readable storage medium, so as to solve the technical defects in the prior art.
According to a first aspect of embodiments herein, there is provided a data acquisition system comprising:
an application server, a remote server;
the application server is configured to establish a data transmission channel with the remote server, send registration information of a data storage module and identification information of an index to be detected to the remote server through the data transmission channel, collect index data of the index to be detected, and store the index data in the data storage module;
and the remote server is configured to call a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information, process the index data according to a preset data processing strategy and display a generated processing result.
According to a second aspect of the embodiments of the present specification, there is provided a data acquisition method applied to a remote server, including:
establishing a data transmission channel with an application server, and receiving registration information of a data storage module and identification information of an index to be detected, which are sent by the application server through the data transmission channel;
calling a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information;
and processing the index data according to a preset data processing strategy, and displaying a generated processing result.
Optionally, the establishing a data transmission channel with an application server includes:
receiving a data transmission channel establishment request sent by an application server;
and establishing a long connection control channel and a long connection data channel with the application server based on the data channel establishing request.
Optionally, the receiving, by the application server, the registration information of the data storage module and the identification information of the indicator to be detected, which are sent through the data transmission channel, includes:
and receiving the registration information of the data storage module and the identification information of the index to be detected, which are sent by the application server through the long connection control channel.
Optionally, the invoking the remote direct memory access module reads the index data from the data storage module according to the registration information and the identification information, including:
and calling a remote direct memory access module to read the index data from the data storage module through the long connection data channel according to the registration information and the identification information.
Optionally, the invoking the remote direct memory access module reads the index data from the data storage module according to the registration information and the identification information, including:
calling a remote direct memory access module to read the index data from the data storage module according to a preset time period;
correspondingly, the processing the index data according to a preset data processing strategy includes:
and calculating and displaying the change results of the indexes to be detected in different time periods based on the index data of the indexes to be detected.
Optionally, the invoking the remote direct memory access module reads the index data from the data storage module according to the registration information and the identification information, including:
calling a remote direct memory access module to read data contained in a data block from the data storage module according to the registration information and the identification information;
analyzing the data contained in the data block to obtain encrypted data and an encryption algorithm identifier;
and determining a decryption algorithm corresponding to the encryption algorithm according to the encryption algorithm identification, and decrypting the encrypted data by adopting the decryption algorithm to generate the index data.
Optionally, the data acquisition method further includes:
and in response to a data acquisition ending instruction sent by the application server, stopping reading the index data from the data storage module, wherein the application server calls the proxy module to release the memory corresponding to the data storage module.
According to a third aspect of embodiments herein, there is provided a data acquisition apparatus comprising:
the receiving module is configured to establish a data transmission channel with an application server and receive registration information of the data storage module and identification information of the index to be detected, which are sent by the application server through the data transmission channel;
the reading module is configured to call a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information;
and the display module is configured to process the index data according to a preset data processing strategy and display the generated processing result.
According to a fourth aspect of the embodiments of the present specification, there is provided a data acquisition method applied to an application server, including:
establishing a data transmission channel with a remote server;
sending registration information of a data storage module and identification information of the index to be detected to the remote server through the data transmission channel;
and acquiring index data of the index to be detected, and storing the index data in the data storage module, wherein the remote server calls a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information.
Optionally, the establishing a data transmission channel with a remote server includes:
and sending a data channel establishing request to the remote server, and establishing a long connection control channel and a long connection data channel with the remote server.
Optionally, the sending, to the remote server through the data transmission channel, the registration information of the data storage module and the identification information of the indicator to be detected includes:
establishing connection between the application program interface and the agent module;
calling an application program interface to determine memory allocation information corresponding to the data storage module through a memory mapping function, and sending the memory allocation information to the proxy module;
and calling the proxy module to send the memory allocation information to the remote server through the long connection control channel.
Optionally, the storing the index data in the data storage module includes:
determining an encryption algorithm, and encrypting the index data by adopting the encryption algorithm;
and storing the encryption result and the encryption algorithm identification in the data storage module in the form of a data block.
Optionally, the data acquisition method further includes:
calling an application program interface to release the index data of the index to be detected;
and calling the proxy module to send a data acquisition ending instruction to the remote server, and releasing the memory corresponding to the data storage module by the proxy module.
Optionally, after the acquiring the index data of the index to be detected, the method further includes:
and the calling proxy module processes the index data according to a first data processing strategy and stores the generated intermediate data in the data storage module, wherein the remote server calls a remote direct memory access module to read the intermediate data from the data storage module according to the registration information and the identification information.
According to a fifth aspect of embodiments herein, there is provided a data acquisition apparatus comprising:
a transmission channel establishing module configured to establish a data transmission channel with a remote server;
the sending module is configured to send the registration information of the data storage module and the identification information of the index to be detected to the remote server through the data transmission channel;
and the acquisition module is configured to acquire the index data of the index to be detected and store the index data in the data storage module, wherein the remote server calls a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information.
According to a sixth aspect of embodiments herein, there is provided a computing device comprising:
a memory and a processor;
the memory is configured to store computer-executable instructions and the processor is configured to execute the computer-executable instructions to implement the steps of the data acquisition method.
According to a seventh aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data acquisition method.
In one embodiment of the specification, a data transmission channel is established between an application server and a remote server, registration information of a data storage module and identification information of an index to be detected are sent to the remote server through the data transmission channel, index data of the index to be detected are collected, and the index data are stored in the data storage module; and the remote server reads the index data from the data storage module according to the registration information and the identification information by calling a remote direct memory access module, processes the index data according to a preset data processing strategy and displays a generated processing result.
In the embodiment of the specification, the remote server reads the index data of the index to be detected in the application server by using a Remote Direct Memory Access (RDMA) module, and since RDMA can directly and quickly transmit the index data from the data storage module of the application server to the remote server, an operating system and a CPU in an application program are not required to participate in the whole process of data transmission, on one hand, the interference on the operating system is favorably reduced, and the jitter brought to the operating system is avoided; on the other hand, the method is beneficial to reducing the consumption of the CPU of the application server in the data transmission process, thereby being beneficial to improving the processing performance of the system and reducing the network communication delay.
Drawings
FIG. 1 is an architecture diagram of a data acquisition system provided in one embodiment of the present disclosure;
FIG. 2 is a flow chart of a data collection method provided in an embodiment of the present disclosure;
FIG. 3 is a schematic diagram of a data acquisition device provided in one embodiment of the present description;
FIG. 4 is a process flow diagram of a data collection method provided in one embodiment of the present description;
FIG. 5 is a schematic diagram of a data acquisition device provided in one embodiment of the present description;
fig. 6 is a block diagram of a computing device according to an embodiment of the present disclosure.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, as those skilled in the art will be able to make and use the present disclosure without departing from the spirit and scope of the present disclosure.
The terminology used in the description of the one or more embodiments is for the purpose of describing the particular embodiments only and is not intended to be limiting of the description of the one or more embodiments. As used in one or more embodiments of the present specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It will be understood that, although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can also be referred to as a second and, similarly, a second can also be referred to as a first without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
First, the noun terms to which one or more embodiments of the present specification relate are explained.
RDMA: in the field of data centers, Remote Direct Memory Access (RDMA) is a technology for accessing data in a memory by bypassing a kernel of a remote host operating system, and since the RDMA does not pass through the operating system, a large amount of CPU resources are saved, the system throughput is also improved, the network communication delay of the system is reduced, and the RDMA is particularly suitable for being widely applied to large-scale parallel computer clusters.
RoCE: RDMA over Converged Ethernet (RoCE) is a network protocol that allows Remote Direct Memory Access (RDMA) over an Ethernet network.
And (3) detection: a method for judging the current running state of an application server by collecting various data of an operating system or an application process of the application server. The application server detection program is deployed on almost all machines needing to provide services to the outside.
A variety of detection systems are widely deployed on current application servers, and these detection systems collect various detection data and store the data locally or upload the data to some database or log server. An application server often deploys a plurality of detection items, and a data acquisition program for executing the detection items increases resource consumption of the application server. Specifically, index data generated by an application process in an application server in the running process can be stored in a memory allocated to the application process, when a remote server needs to read the index data in the memory of the application process, the remote server needs to communicate with the application server, that is, a data acquisition request is sent to the application server, and after receiving the data acquisition request, the application server reads the index data in the memory and sends the index data to the remote server, which brings CPU overhead to the application server.
At present, RDMA/RoCE is widely applied to a data center network as a communication mode bypassing an operating system kernel, and application scenes of the RDMA/RoCE are more and more extensive with continuous improvement of hardware performance and application performance. RDMA can support direct remote memory access, and can directly read and write the memory of the application server on the remote server based on the RDMA technology without the intervention of the CPU of the application server, so that almost zero CPU overhead of the application server can be realized.
Therefore, the embodiments of the present specification combine RDMA technology in a detection system to implement a method for remote data acquisition based on RDMA. The method is beneficial to reducing the interference to the CPU of the application server in the process of data transmission between the application server and the remote server, thereby reducing the consumption of the CPU of the application server in the data transmission process, improving the processing performance of the system and being beneficial to reducing the network communication delay.
In the present specification, a data acquisition system is provided, and the present specification relates to a data acquisition method, a data acquisition apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
Fig. 1 shows an architecture diagram of a data acquisition system provided according to an embodiment of the present specification, including:
an application server 102 and a remote server 104;
the application server 102 is configured to establish a data transmission channel with the remote server 104, send registration information of a data storage module and identification information of an index to be detected to the remote server through the data transmission channel, collect index data of the index to be detected, and store the index data in the data storage module;
the remote server 104 is configured to invoke a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information, process the index data according to a preset data processing strategy, and display a generated processing result.
Specifically, an application Server (App Server), that is, a Server where an application process is located, is also a Server that needs to acquire detection data of an index to be detected of the application process; and the Remote Server (Remote Collection Server) is responsible for acquiring the required index data of the index to be detected from each application Server through RDMA Read.
An application program interface (SDK interface) is a software development kit provided by the RDMA-based detection system, and can provide a series of simple and easy-to-use functions and methods for accelerating the application to be quickly accessed to the RDMA-based detection system; the agent module (Zagent, zero-overhead agent) is used as an acquisition process running on the application server and is responsible for communicating with the remote server, and some required information on the application server is uploaded to the remote server, so that the remote server can directly acquire and analyze data through RDMA Read.
The data storage module (Share Memory), i.e. the shared Memory, is a storage area independent from the application process, and stores therein index data to be acquired by the remote server. This memory is shared between the agent and the application process or between the agent and the operating system kernel through shared memory techniques. The proxy module registers the RDMA network card and uploads the information such as the memory address, the size and the like after the registration to a remote server.
In this embodiment of the present specification, a process of acquiring, by a remote server, index data of an index to be detected of an application process of an application server based on RDMA is specifically as follows:
an initialization stage:
after an operating system in an application server starts to run, an agent module enters an initialization process, and in the initialization process, the application server establishes connection with a remote server, namely a long connection control channel and a long connection data channel are established; then the application process runs and initializes, the application process calls an application program interface to establish connection with the proxy module so as to transmit necessary memory related information through the connection; then the application program interface applies for a data storage module storing index data of the index to be detected for the application process through a mmap function (memory mapping function), namely, a memory is allocated for the index data of the index to be detected of the application process; after the application is successful, initializing identification information of the to-be-detected indexes needing to be detected in the application process, such as determining positions of the corresponding indexes needing to be stored, storage formats of the indexes and the like; the mmap function is used for establishing mapping from a virtual space to a disk space, a virtual space address can be mapped to a disk file, when the address is not set, the system automatically sets the address, the function returns a corresponding memory address (virtual address), when the address is accessed, the content on the disk needs to be copied to a memory, then the memory can be read or written, finally, the data on the memory can be changed back to the disk through a manmap function, namely, the mapping between the virtual space and the memory space is removed, and the process of process sharing of data (shared memory) can be realized through the mode.
The application program interface sends memory allocation information of a corresponding data storage module and identification information of the index to be detected to the proxy module, the proxy module calls an RDMA related interface to perform Read-only registration on the data storage module, and the registration can prevent a physical memory corresponding to the data storage module from being allocated, migrated, destroyed or released, so that a subsequent RDMA network card can directly Read index data stored in the data storage module through RDMA Read;
after the agent module finishes registering, the registration information of the data storage module and identification information related to some indexes to be detected of the application process are sent to the remote server through the long connection control channel, so that after initialization is finished, the remote server can directly Read the index data of the indexes to be detected, which need to be detected by the application process, through RDMA Read.
A data reading stage:
the remote server determines the acquisition frequency of the index data according to the configuration of the remote server and the identification information of the index to be detected uploaded by the proxy module, and determines which data storage module to acquire the index data stored by the data storage module according to the identification information of the data storage module, the process is actively initiated by the remote server, the RDMA Read is used for acquiring the index data in the data storage module, and then calculation can be carried out based on the index data, and the calculation result is displayed.
And (3) a release stage:
when the application process exits, the index data of the index to be detected of the application process stops collecting, and firstly, the application process calls an application program interface to release the index data of the index to be detected; or under the condition that the application process is abnormally quitted, the operating system recovers the memory corresponding to the data storage module; after sensing that the application process exits, the agent module informs the remote server to stop collecting the index data of the to-be-detected index of the application process; and after receiving the instruction of stopping acquisition, the remote server confirms that the acquisition of the index data is stopped, and then the agent module is the memory resource of the data storage module registered by the application process. To this end, the initialization-data acquisition-release phase of the application process ends.
In the embodiment of the description, data acquisition is periodically performed on the remote server in an RDMA Read manner, so that additional consumption of CPU/memory/Read-write resources of the application server is not required, the influence on the application process is reduced, and the transmission efficiency of data transmission from the application server to the remote server is improved.
In one embodiment of the specification, a data transmission channel is established between an application server and a remote server, registration information of a data storage module and identification information of an index to be detected are sent to the remote server through the data transmission channel, index data of the index to be detected are collected, and the index data are stored in the data storage module; and the remote server reads the index data from the data storage module according to the registration information and the identification information by calling a remote direct memory access module, processes the index data according to a preset data processing strategy and displays a generated processing result.
In the embodiment of the specification, the remote server reads the index data of the index to be detected in the application server by using a Remote Direct Memory Access (RDMA) module, and since RDMA can directly and quickly transmit the index data from the data storage module of the application server to the remote server, an operating system and a CPU in an application program are not required to participate in the whole process of data transmission, on one hand, the interference on the operating system is favorably reduced, and the jitter brought to the operating system is avoided; on the other hand, the method is beneficial to reducing the consumption of the CPU of the application server in the data transmission process, thereby being beneficial to improving the processing performance of the system and reducing the network communication delay.
Fig. 2 shows a process flow diagram of a data acquisition method provided in accordance with one embodiment of the present specification, including steps 202 through 206.
Step 202, establishing a data transmission channel with an application server, and receiving registration information of a data storage module and identification information of an index to be detected, which are sent by the application server through the data transmission channel.
Specifically, the data collection method provided in the embodiments of the present specification is applied to a Remote Server (Remote collection Server), and is responsible for collecting the index data of the to-be-detected index of the application process from each application Server by calling a Remote direct memory access module (RDMA).
Before the remote server reads the index data of the to-be-detected index of the application process from the application server, the application server initializes the application process and a memory for storing the index data of the to-be-detected index of the application process.
Specifically, after an operating system in the application server starts to run, a proxy module (zero-overhead agent) in the application server enters an initialization process, and in the initialization process, the application server establishes a connection with the remote server, and the proxy module is used as an acquisition process running on the application server and is responsible for communicating with the remote server through the connection, and some required information on the application server is uploaded to the remote server, so that the remote server can directly acquire and analyze data through RDMA Read.
In addition, an application program interface is also arranged in the application server, and an application process calls the application program interface to establish connection with the proxy module so as to transmit necessary memory related information through the connection; then, the application program interface applies for a data storage module storing index data of the index to be detected for the application process through a mmap function (memory mapping function), that is, a memory is allocated for the index data of the index to be detected for the application process, and the memory specifically includes the position of the memory, the size of the memory, and the like; after the application is successful, initializing identification information of the to-be-detected indexes needing to be detected in the application process, such as determining positions of the corresponding indexes needing to be stored, storage formats of the indexes and the like;
then the application program interface sends the memory allocation information of the corresponding data storage module and the identification information of the index to be detected to the proxy module, the proxy module calls the RDMA related interface to perform Read-only registration on the data storage module, and the registration can prevent the physical memory corresponding to the data storage module from being allocated, migrated, destroyed or released, so that a subsequent RDMA network card can directly Read the index data stored in the data storage module through RDMA Read;
after the registration is finished, the agent module sends the registration information of the data storage module and identification information related to some indexes to be detected of the application process to the remote server through the data transmission channel, so that after the initialization is finished, the remote server can directly Read the index data of the indexes to be detected, which need to be detected by the application process, through RDMA Read.
In specific implementation, a data transmission channel is established with an application server, specifically, a data transmission channel establishment request sent by the application server is received, and a long connection control channel and a long connection data channel are established with the application server based on the data channel establishment request.
Further, the registration information of the data storage module and the identification information of the index to be detected, which are sent by the application server through the data transmission channel, are received, that is, the registration information of the data storage module and the identification information of the index to be detected, which are sent by the application server through the long connection control channel, are received.
Specifically, the data transmission channel consists of a long connection control channel and a long connection data channel, so that in the process of establishing the data transmission channel between the application server and the remote server, the application server sends a channel establishment request to the remote server, the long connection control channel and the long connection data channel between the application server and the remote server are established in a long connection mode, and the proxy module of the application server sends the registration information of the data storage module and the identification information of the index to be detected to the remote server through the long connection control channel.
By establishing the data transmission channel and utilizing the data transmission channel to transmit information, the data transmission efficiency is favorably improved, and the data acquisition efficiency is improved.
And 204, calling a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information.
In specific implementation, the remote direct memory access module is called to read the index data from the data storage module according to the registration information and the identification information, specifically, the remote direct memory access module is called to read the index data from the data storage module through the long connection data channel according to the registration information and the identification information.
Specifically, after receiving the identification information of the index to be detected uploaded by the proxy module, the remote server can determine the acquisition frequency of the index data according to the configuration of the remote server and the identification information, and determine which data storage module to acquire the index data stored in according to the identification information of the data storage module, wherein the process is actively initiated by the remote server, and the RDMA Read is used for acquiring the index data in the data storage module.
Or, invoking a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information, and can be realized by the following steps:
calling a remote direct memory access module to read data contained in a data block from the data storage module according to the registration information and the identification information;
analyzing the data contained in the data block to obtain encrypted data and an encryption algorithm identifier;
and determining a decryption algorithm corresponding to the encryption algorithm according to the encryption algorithm identification, and decrypting the encrypted data by adopting the decryption algorithm to generate the index data.
Specifically, in order to ensure the security of the index data transmitted between the application server and the remote server, in the embodiment of the present specification, after the application server collects the index data of the index to be detected, a preset encryption algorithm is used to encrypt the index data to generate a corresponding encryption result, and the encryption result and the algorithm identifier of the encryption algorithm are stored in the data storage module in the form of a data block.
The remote server can directly read the data contained in the data block from the data storage module and analyze the data contained in the data block to obtain the encrypted data and the encrypted algorithm identification so as to determine the decryption algorithm corresponding to the encryption algorithm according to the encrypted algorithm identification and decrypt the encrypted data by adopting the decryption algorithm, because the remote server needs to process the index data according to a preset data processing strategy after reading the index data of the index to be detected in the application process in the application server, the remote server can directly read the data contained in the data block from the data storage module after encrypting the index data by the application server and storing the encrypted result and the encrypted algorithm identification to the data storage module in the form of the data block, and generating the index data.
In practical applications, the Encryption algorithm may be a symmetric Encryption algorithm, including but not limited to DES (Data Encryption Standard) algorithm, 3DES algorithm, AES (Advanced Encryption Standard) algorithm, TDEA algorithm, Blowfish algorithm, RC5 algorithm, IDEA algorithm, and the like.
And step 206, processing the index data according to a preset data processing strategy, and displaying the generated processing result.
Specifically, different indexes to be detected may correspond to different preset data processing strategies, and may be determined specifically according to actual needs, which is not limited herein. In practical application, a data processing strategy of the index to be detected can be determined according to the project applied by the data acquisition method, the index data of the index to be detected is processed according to the data processing strategy, and a corresponding processing result is generated and displayed.
For example, if the index to be detected is a CPU, the usage rate of the CPU in one day needs to be calculated, or the usage rates of the CPU in the first time interval and the second time interval are calculated, and the increase rate or the decrease rate of the usage rate of the CPU in the second time interval relative to the usage rate of the CPU in the first time interval is further calculated; or when the data acquisition method is applied to a transaction scene, the indexes to be detected can be transaction amount, transaction time, transaction types and the like, and the data processing strategy can be to calculate the transaction amount of the target transaction object in one day, calculate the transaction amount of the target transaction object in different time periods in one day, or calculate the transaction amount of the target transaction object in a first time period and a second time period, and calculate the increase rate or decrease rate of the transaction amount of the target transaction object in the second time period relative to the transaction amount in the first time period.
In addition, the remote direct memory access module is called to read the index data from the data storage module according to the registration information and the identification information, specifically, the remote direct memory access module is called to read the index data from the data storage module according to a preset time period;
correspondingly, the index data are processed according to a preset data processing strategy, that is, based on the index data of the index to be detected, the change results of the index to be detected in different time periods are calculated and displayed.
Specifically, the remote server may periodically acquire the index data of the to-be-detected index from the data storage module of the application server, for example, the index data is acquired every 10 seconds, and if the index data of the to-be-detected index changes within the 10 seconds, the remote server may calculate a change rate (an increase rate or a decrease rate) of the index value of the to-be-detected index within the 10 seconds according to the index data acquired twice, or count a relevant statistical value of the to-be-detected index according to the index data acquired twice, and then display a calculation result or a statistical result.
In addition, in addition to reading the index data from the application server by the remote server directly using RDMA in the foregoing manner and processing the index data according to the preset data processing policy to obtain the corresponding processing result, the embodiment of the present specification may also reserve part of the calculation work of the proxy module.
For example, after the application server stores the index data of the to-be-detected index of the application process in the data storage module, the agent module may also read the index data from the data storage module according to a preset time period and calculate the index data, a data processing policy used in the calculation process may be the same as or different from a preset data policy used by the remote server, and a calculation logic that can be provided by a specific agent module may be specifically determined according to actual requirements.
After the calculation is finished, the calculation result is stored in the data storage module, and in this case, when the remote server reads data, the calculation result stored in the data storage module can be directly read, so that the data volume of the data required to be read by the remote server is reduced, and the calculation volume in the remote server is reduced.
In addition, under the condition that the application process of the application server exits, a data acquisition ending instruction can be sent to the remote server, the application server responds to the data acquisition ending instruction sent by the application server and stops reading the index data from the data storage module, and the application server calls the proxy module to release the memory corresponding to the data storage module.
Specifically, when the application process of the application server exits, the acquisition of the index data of the index to be detected of the application process is stopped, and the application process calls an application program interface to release the index data of the index to be detected; or under the condition that the application process is abnormally quitted, the operating system recovers the memory corresponding to the data storage module; after sensing that the application process exits, the agent module informs the remote server to stop collecting the index data of the to-be-detected index of the application process; and after receiving the instruction of stopping acquisition, the remote server confirms that the acquisition of the index data is stopped, and then the agent module is the memory resource of the data storage module registered by the application process. And ending the process of acquiring the index data of the index to be detected of the application process by the remote server.
In specific implementation, more acquisition items besides the application program, the traffic detection and the core data can be added to the front end of the application program interface, or most of data detection acquisition work can be unloaded to a remote server, but part of calculation work of the proxy module is still reserved.
In the embodiment of the description, except for the initialization and resource release stages, the proxy module synchronizes the relevant information of the data storage module to the remote server, the two key steps of index data acquisition and data uploading are completed by the remote server, the expenses on an application server CPU, a memory, an I/O (input/output) and the like only exist in the initialization stage, and the influence on the application server is small.
The embodiment of the specification combines the RDMA technology with data acquisition, and the data detection and acquisition process is unloaded to the remote server by utilizing the capacity of remote memory reading and writing of the RDMA, so that the CPU overhead of the application server caused by data detection and acquisition transmission is avoided.
In one embodiment of the specification, a data transmission channel is established between an application server and a remote server, registration information of a data storage module and identification information of an index to be detected are sent to the remote server through the data transmission channel, index data of the index to be detected are collected, and the index data are stored in the data storage module; and the remote server reads the index data from the data storage module according to the registration information and the identification information by calling a remote direct memory access module, processes the index data according to a preset data processing strategy and displays a generated processing result.
In the embodiment of the specification, the remote server reads the index data of the index to be detected in the application server by using a Remote Direct Memory Access (RDMA) module, and since RDMA can directly and quickly transmit the index data from the data storage module of the application server to the remote server, an operating system and a CPU in an application program are not required to participate in the whole process of data transmission, on one hand, the interference on the operating system is favorably reduced, and the jitter brought to the operating system is avoided; on the other hand, the method is beneficial to reducing the consumption of the CPU of the application server in the data transmission process, thereby being beneficial to improving the processing performance of the system and reducing the network communication delay.
Corresponding to the above method embodiment, the present specification further provides a data acquisition device embodiment, and fig. 3 shows a schematic diagram of a data acquisition device provided in an embodiment of the present specification. As shown in fig. 3, the apparatus includes:
a receiving module 302, configured to establish a data transmission channel with an application server, and receive registration information of a data storage module and identification information of an index to be detected, which are sent by the application server through the data transmission channel;
a reading module 304, configured to invoke a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information;
and the display module 306 is configured to process the index data according to a preset data processing strategy and display the generated processing result.
Optionally, the receiving module 302 includes:
the first receiving submodule is configured to receive a data transmission channel establishment request sent by an application server;
a channel establishing submodule configured to establish a long connection control channel and a long connection data channel with the application server based on the data channel establishing request.
Optionally, the receiving module 302 includes:
and the second receiving submodule is configured to receive the registration information of the data storage module and the identification information of the index to be detected, which are sent by the application server through the long connection control channel.
Optionally, the reading module 304 includes:
and the first reading submodule is configured to call a remote direct memory access module to read the index data from the data storage module through the long connection data channel according to the registration information and the identification information.
Optionally, the reading module 304 includes:
the second reading submodule is configured to call a remote direct memory access module to read the index data from the data storage module according to a preset time period;
accordingly, the display module 306 includes:
and the display submodule is configured to calculate and display the change result of the to-be-detected index in different time periods based on the index data of the to-be-detected index.
Optionally, the reading module 304 includes:
the third reading submodule is configured to call a remote direct memory access module to read data contained in a data block from the data storage module according to the registration information and the identification information;
the analysis submodule is configured to analyze data contained in the data block to obtain encrypted data and an encryption algorithm identifier;
and the decryption submodule is configured to determine a decryption algorithm corresponding to the encryption algorithm according to the encryption algorithm identifier, and decrypt the encrypted data by using the decryption algorithm to generate the index data.
Optionally, the data acquisition device further includes:
and the response module is configured to respond to a data acquisition ending instruction sent by the application server and stop reading the index data from the data storage module, wherein the application server calls the proxy module to release the memory corresponding to the data storage module.
The above is a schematic configuration of a data acquisition apparatus of the present embodiment. It should be noted that the technical solution of the data acquisition apparatus and the technical solution of the data acquisition method belong to the same concept, and details that are not described in detail in the technical solution of the data acquisition apparatus can be referred to the description of the technical solution of the data acquisition method.
Fig. 4 shows a process flow diagram of a data acquisition method provided in accordance with one embodiment of the present specification, including steps 402 to 406.
Step 402, establishing a data transmission channel with a remote server.
Specifically, the data acquisition method provided in the embodiments of the present specification is applied to an application Server (App Server), where the application Server is a Server that needs to acquire detection data of an indicator to be detected in an application process, and a Remote Server (Remote collection Server) is responsible for acquiring the indicator data of the required indicator to be detected from each application Server through RDMA Read; after the application server collects the index data of the index to be detected in the application process, the remote server reads the index data from the application server by using the RDMA technology.
Before the remote server reads the index data of the to-be-detected index of the application process from the application server, the application server needs to initialize the application process and a memory for storing the index data of the to-be-detected index of the application process, and the specific initialization process may refer to the initialization process described in the foregoing method embodiment applied to the remote server, which is not described herein again.
In the initialization process, the application server establishes connection with the remote server, namely, establishes a data transmission channel, and during specific implementation, establishes the data transmission channel with the remote server, and the method comprises the following steps: and sending a data channel establishing request to the remote server, and establishing a long connection control channel and a long connection data channel with the remote server.
Step 404, sending the registration information of the data storage module and the identification information of the index to be detected to the remote server through the data transmission channel.
In specific implementation, the sending, to the remote server, the registration information of the data storage module and the identification information of the to-be-detected indicator through the data transmission channel includes:
establishing connection between the application program interface and the agent module;
calling an application program interface to determine memory allocation information corresponding to the data storage module through a memory mapping function, and sending the memory allocation information to the proxy module;
and calling the proxy module to send the memory allocation information to the remote server through the long connection control channel.
Specifically, an application program interface, namely an SDK interface, which is a software development kit provided by the RDMA-based detection system, can provide a series of simple and easy-to-use functions and methods for accelerating an application to quickly access the RDMA-based detection system; the agent module (Zagent, zero-overhead agent) is used as an acquisition process running on the application server and is responsible for communicating with the remote server, and some required information on the application server is uploaded to the remote server, so that the remote server can directly acquire and analyze data through RDMA Read.
And 406, acquiring index data of the index to be detected, and storing the index data in the data storage module, wherein the remote server calls a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information.
Specifically, the data storage module (Share Memory), i.e., the shared Memory, stores therein index data to be collected by the remote server. This memory is shared between the agent and the application process or between the agent and the operating system kernel through shared memory techniques. The proxy module registers the RDMA network card and uploads the information such as the memory address, the size and the like after the registration to a remote server.
The remote server can call a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information, specifically, the remote direct memory access module is called to read the index data from the data storage module through the long connection data channel according to the registration information and the identification information.
In specific implementation, the storing the index data in the data storage module includes:
determining an encryption algorithm, and encrypting the index data by adopting the encryption algorithm;
and storing the encryption result and the encryption algorithm identification in the data storage module in the form of a data block.
Specifically, in order to ensure the security of the index data transmitted between the application server and the remote server, in the embodiment of the present specification, after the application server collects the index data of the index to be detected, a preset encryption algorithm is used to encrypt the index data to generate a corresponding encryption result, and the encryption result and the algorithm identifier of the encryption algorithm are stored in the data storage module in the form of a data block.
The remote server can directly read the encryption result from the data storage module of the application server by using the RDMA technology under the condition that the data needs to be read from the application server, but because the remote server needs to process the index data according to a preset data processing strategy after reading the index data of the index to be detected in the application process of the application server, the remote server can directly read the data in the data block from the data storage module after encrypting the index data by the application server and storing the encryption result and the encryption algorithm identification to the data storage module in the form of the data block, and analyze the data in the data block to obtain the encryption data and the encryption algorithm identification so as to determine the decryption algorithm corresponding to the encryption algorithm according to the encryption algorithm identification and decrypt the encryption data by adopting the decryption algorithm, and generating the index data.
In practical applications, the Encryption algorithm may be a symmetric Encryption algorithm, including but not limited to DES (Data Encryption Standard) algorithm, 3DES algorithm, AES (Advanced Encryption Standard) algorithm, TDEA algorithm, Blowfish algorithm, RC5 algorithm, IDEA algorithm, and the like.
In specific implementation, when an application process of an application server exits, an application program interface can be called to release the index data of the index to be detected, a proxy module is called to send a data acquisition ending instruction to the remote server, and the proxy module releases a memory corresponding to the data storage module.
Specifically, when the application process of the application server exits, the acquisition of the index data of the index to be detected of the application process is stopped, and the application process calls an application program interface to release the index data of the index to be detected; or under the condition that the application process is abnormally quitted, the operating system recovers the memory corresponding to the data storage module; after sensing that the application process exits, the agent module informs the remote server to stop collecting the index data of the to-be-detected index of the application process; and after receiving the instruction of stopping acquisition, the remote server confirms that the acquisition of the index data is stopped, and then the agent module is the memory resource of the data storage module registered by the application process. And ending the process of acquiring the index data of the index to be detected of the application process by the remote server.
In specific implementation, after the index data of the index to be detected is collected, the method further includes:
and the calling proxy module processes the index data according to a first data processing strategy and stores the generated intermediate data in the data storage module, wherein the remote server calls a remote direct memory access module to read the intermediate data from the data storage module according to the registration information and the identification information.
In particular, embodiments of the present description may offload a majority of the data detection acquisition work to a remote server, while still retaining a portion of the computing work of the proxy module.
In one embodiment of the specification, a data transmission channel is established between an application server and a remote server, registration information of a data storage module and identification information of an index to be detected are sent to the remote server through the data transmission channel, index data of the index to be detected are collected, and the index data are stored in the data storage module; and the remote server reads the index data from the data storage module according to the registration information and the identification information by calling a remote direct memory access module, processes the index data according to a preset data processing strategy and displays a generated processing result.
In the embodiment of the specification, the remote server reads the index data of the index to be detected in the application server by using a Remote Direct Memory Access (RDMA) module, and since RDMA can directly and quickly transmit the index data from the data storage module of the application server to the remote server, an operating system and a CPU in an application program are not required to participate in the whole process of data transmission, on one hand, the interference on the operating system is favorably reduced, and the jitter brought to the operating system is avoided; on the other hand, the method is beneficial to reducing the consumption of the CPU of the application server in the data transmission process, thereby being beneficial to improving the processing performance of the system and reducing the network communication delay.
Corresponding to the above method embodiment, the present specification further provides an embodiment of a data acquisition device, and fig. 5 shows a schematic diagram of a data acquisition device provided in an embodiment of the present specification. As shown in fig. 5, the apparatus includes:
a transmission channel establishing module 502 configured to establish a data transmission channel with a remote server;
a sending module 504 configured to send registration information of a data storage module and identification information of an index to be detected to the remote server through the data transmission channel;
the acquisition module 506 is configured to acquire the index data of the index to be detected and store the index data in the data storage module, wherein the remote server calls a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information.
Optionally, the transmission channel establishing module 502 includes:
and the sending submodule is configured to send a data channel establishing request to the remote server and establish a long connection control channel and a long connection data channel with the remote server.
Optionally, the sending module 504 includes:
the connection establishment submodule is configured to establish connection between the application program interface and the agent module;
the first sending submodule is configured to call an application program interface to determine memory allocation information corresponding to the data storage module through a memory mapping function, and send the memory allocation information to the agent module;
and the second sending submodule is configured to call the proxy module to send the memory allocation information to the remote server through the long connection control channel.
Optionally, the acquisition module 506 includes:
an encryption submodule configured to determine an encryption algorithm with which the index data is encrypted;
and the storage submodule is configured to store the encryption result and the encryption algorithm identification in the data storage module in the form of a data block.
Optionally, the data acquisition device further includes:
the calling module is configured to call an application program interface to release the index data of the index to be detected;
and the instruction sending module is configured to call the proxy module to send a data acquisition ending instruction to the remote server, and the proxy module releases the memory corresponding to the data storage module.
Optionally, the data acquisition device further includes:
and the processing module is configured to call the proxy module to process the index data according to a first data processing strategy and store the generated intermediate data in the data storage module, wherein the remote server calls the remote direct memory access module to read the intermediate data from the data storage module according to the registration information and the identification information.
The above is a schematic configuration of a data acquisition apparatus of the present embodiment. It should be noted that the technical solution of the data acquisition apparatus and the technical solution of the data acquisition method belong to the same concept, and details that are not described in detail in the technical solution of the data acquisition apparatus can be referred to the description of the technical solution of the data acquisition method.
FIG. 6 illustrates a block diagram of a computing device 600 provided in accordance with one embodiment of the present description. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is coupled to the memory 610 via a bus 630 and a database 650 is used to store data.
Computing device 600 also includes access device 640, access device 640 enabling computing device 600 to communicate via one or more networks 660. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. Access device 640 may include one or more of any type of network interface (e.g., a Network Interface Card (NIC)) whether wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.
In one embodiment of the present description, the above-described components of computing device 600, as well as other components not shown in FIG. 6, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device architecture shown in FIG. 6 is for purposes of example only and is not limiting as to the scope of the present description. Those skilled in the art may add or replace other components as desired.
Computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smartphone), wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 600 may also be a mobile or stationary server.
Wherein the memory 610 is configured to store computer-executable instructions and the processor 620 is configured to execute the following computer-executable instructions for implementing the steps of the data acquisition method.
The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the data acquisition method belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the data acquisition method.
An embodiment of the present specification also provides a computer readable storage medium storing computer instructions, which when executed by a processor, are used for implementing the steps of the data acquisition method.
The above is an illustrative scheme of a computer-readable storage medium of the present embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the data acquisition method, and details that are not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the data acquisition method.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts, but those skilled in the art should understand that the present embodiment is not limited by the described acts, because some steps may be performed in other sequences or simultaneously according to the present embodiment. Further, those skilled in the art should also appreciate that the embodiments described in this specification are preferred embodiments and that acts and modules referred to are not necessarily required for an embodiment of the specification. In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
The preferred embodiments of the present specification disclosed above are intended only to aid in the description of the specification. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the embodiments and the practical application, to thereby enable others skilled in the art to best understand and utilize the embodiments. The specification is limited only by the claims and their full scope and equivalents.

Claims (14)

1. A data acquisition system comprising:
an application server, a remote server;
the application server is configured to establish a data transmission channel with the remote server, send registration information of a data storage module and identification information of an index to be detected to the remote server through the data transmission channel, collect index data of the index to be detected, and store the index data in the data storage module;
and the remote server is configured to call a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information, process the index data according to a preset data processing strategy and display a generated processing result.
2. A data acquisition method is applied to a remote server and comprises the following steps:
establishing a data transmission channel with an application server, and receiving registration information of a data storage module and identification information of an index to be detected, which are sent by the application server through the data transmission channel;
calling a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information;
and processing the index data according to a preset data processing strategy, and displaying a generated processing result.
3. The data acquisition method of claim 2, wherein the establishing a data transmission channel with an application server comprises:
receiving a data transmission channel establishment request sent by an application server;
and establishing a long connection control channel and a long connection data channel with the application server based on the data channel establishing request.
4. The data acquisition method according to claim 3, wherein the receiving registration information of the data storage module and identification information of the indicator to be detected, which are sent by the application server through the data transmission channel, comprises:
and receiving the registration information of the data storage module and the identification information of the index to be detected, which are sent by the application server through the long connection control channel.
5. The data acquisition method of claim 3, wherein the invoking of the remote direct memory access module reads the index data from the data storage module according to the registration information and the identification information, and comprises:
and calling a remote direct memory access module to read the index data from the data storage module through the long connection data channel according to the registration information and the identification information.
6. The data acquisition method of claim 2, wherein the invoking of the remote direct memory access module reads the index data from the data storage module according to the registration information and the identification information, and comprises:
calling a remote direct memory access module to read the index data from the data storage module according to a preset time period;
correspondingly, the processing the index data according to a preset data processing strategy includes:
and calculating and displaying the change results of the indexes to be detected in different time periods based on the index data of the indexes to be detected.
7. A data acquisition device comprising:
the receiving module is configured to establish a data transmission channel with an application server and receive registration information of the data storage module and identification information of the index to be detected, which are sent by the application server through the data transmission channel;
the reading module is configured to call a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information;
and the display module is configured to process the index data according to a preset data processing strategy and display the generated processing result.
8. A data acquisition method is applied to an application server and comprises the following steps:
establishing a data transmission channel with a remote server;
sending registration information of a data storage module and identification information of the index to be detected to the remote server through the data transmission channel;
and acquiring index data of the index to be detected, and storing the index data in the data storage module, wherein the remote server calls a remote direct memory access module to read the index data from the data storage module according to the registration information and the identification information.
9. The data acquisition method of claim 8, wherein establishing a data transmission channel with a remote server comprises:
and sending a data channel establishing request to the remote server, and establishing a long connection control channel and a long connection data channel with the remote server.
10. The data acquisition method according to claim 9, wherein the sending of the registration information of the data storage module and the identification information of the indicator to be detected to the remote server through the data transmission channel comprises:
establishing connection between the application program interface and the agent module;
calling an application program interface to determine memory allocation information corresponding to the data storage module through a memory mapping function, and sending the memory allocation information to the proxy module;
and calling the proxy module to send the memory allocation information to the remote server through the long connection control channel.
11. The data acquisition method of claim 9, the storing the metric data in the data storage module, comprising:
determining an encryption algorithm, and encrypting the index data by adopting the encryption algorithm;
and storing the encryption result and the encryption algorithm identification in the data storage module in the form of a data block.
12. The data acquisition method according to claim 8, further comprising, after acquiring the index data of the index to be detected:
and the calling proxy module processes the index data according to a first data processing strategy and stores the generated intermediate data in the data storage module, wherein the remote server calls a remote direct memory access module to read the intermediate data from the data storage module according to the registration information and the identification information.
13. A computing device, comprising:
a memory and a processor;
the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions to implement the steps of the data acquisition method of any one of claims 2 to 6 or 8 to 12.
14. A computer readable storage medium storing computer instructions which, when executed by a processor, carry out the steps of the data acquisition method of any one of claims 2 to 6 or 8 to 12.
CN202110533607.6A 2021-05-17 2021-05-17 Data acquisition system, method and device Pending CN113297110A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110533607.6A CN113297110A (en) 2021-05-17 2021-05-17 Data acquisition system, method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110533607.6A CN113297110A (en) 2021-05-17 2021-05-17 Data acquisition system, method and device

Publications (1)

Publication Number Publication Date
CN113297110A true CN113297110A (en) 2021-08-24

Family

ID=77322470

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110533607.6A Pending CN113297110A (en) 2021-05-17 2021-05-17 Data acquisition system, method and device

Country Status (1)

Country Link
CN (1) CN113297110A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114691051A (en) * 2022-05-30 2022-07-01 恒生电子股份有限公司 Data processing method and device

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008310486A (en) * 2007-06-13 2008-12-25 Olympus Corp Communication interface device
US20110246598A1 (en) * 2010-04-02 2011-10-06 Microsoft Corporation Mapping rdma semantics to high speed storage
US20170277655A1 (en) * 2016-03-25 2017-09-28 Microsoft Technology Licensing, Llc Memory sharing for working data using rdma
CN110019110A (en) * 2017-07-28 2019-07-16 腾讯科技（深圳）有限公司 A kind of capacity management methods of operation system, device, equipment and operation system
KR20190087783A (en) * 2018-01-17 2019-07-25 한국전자통신연구원 Apparatus for distributed processing through remote direct memory access and method for the same
CN110113420A (en) * 2019-05-08 2019-08-09 重庆大学 Distributed Message Queue management system based on NVM
CN110402568A (en) * 2018-02-24 2019-11-01 华为技术有限公司 A kind of method and device of communication
CN110445826A (en) * 2018-05-04 2019-11-12 阿里巴巴集团控股有限公司 A kind of session information acquisition methods, device and server
US20200042444A1 (en) * 2018-08-02 2020-02-06 MemVerge, Inc Distributed Memory Object Architecture
CN110908951A (en) * 2019-10-30 2020-03-24 浙江科技学院 Distributed high-speed data acquisition system based on network interface
WO2021022794A1 (en) * 2019-08-06 2021-02-11 创新先进技术有限公司 Rdma-based data transmission method, network card, server and medium
CN112612734A (en) * 2020-12-18 2021-04-06 平安科技（深圳）有限公司 File transmission method and device, computer equipment and storage medium

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008310486A (en) * 2007-06-13 2008-12-25 Olympus Corp Communication interface device
US20110246598A1 (en) * 2010-04-02 2011-10-06 Microsoft Corporation Mapping rdma semantics to high speed storage
US20170277655A1 (en) * 2016-03-25 2017-09-28 Microsoft Technology Licensing, Llc Memory sharing for working data using rdma
CN110019110A (en) * 2017-07-28 2019-07-16 腾讯科技（深圳）有限公司 A kind of capacity management methods of operation system, device, equipment and operation system
KR20190087783A (en) * 2018-01-17 2019-07-25 한국전자통신연구원 Apparatus for distributed processing through remote direct memory access and method for the same
CN110402568A (en) * 2018-02-24 2019-11-01 华为技术有限公司 A kind of method and device of communication
CN110445826A (en) * 2018-05-04 2019-11-12 阿里巴巴集团控股有限公司 A kind of session information acquisition methods, device and server
US20200042444A1 (en) * 2018-08-02 2020-02-06 MemVerge, Inc Distributed Memory Object Architecture
CN110113420A (en) * 2019-05-08 2019-08-09 重庆大学 Distributed Message Queue management system based on NVM
WO2021022794A1 (en) * 2019-08-06 2021-02-11 创新先进技术有限公司 Rdma-based data transmission method, network card, server and medium
CN110908951A (en) * 2019-10-30 2020-03-24 浙江科技学院 Distributed high-speed data acquisition system based on network interface
CN112612734A (en) * 2020-12-18 2021-04-06 平安科技（深圳）有限公司 File transmission method and device, computer equipment and storage medium

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114691051A (en) * 2022-05-30 2022-07-01 恒生电子股份有限公司 Data processing method and device

Similar Documents

Publication Publication Date Title
Noor et al. Mobile cloud computing: Challenges and future research directions
Hao et al. Challenges and software architecture for fog computing
US9747093B2 (en) Device driver aggregation in operating system deployment
CN112583882A (en) Methods, systems, articles of manufacture, and apparatus for managing telemetry data in an edge environment
George et al. Mez: An adaptive messaging system for latency-sensitive multi-camera machine vision at the iot edge
WO2020220536A1 (en) Data backup method and device, and computer readable storage medium
CN112486074B (en) Data processing system, method and device
CN111163130A (en) Network service system and data transmission method thereof
CN108933801B (en) Method and device for establishing cloud desktop channel and cloud desktop communication
US11770458B1 (en) Systems for exchanging data using intermediate devices
CN107102889B (en) Virtual machine resource adjusting method and device
CN113297110A (en) Data acquisition system, method and device
CN114996134A (en) Containerized deployment method, electronic equipment and storage medium
US20200143583A1 (en) Cloud render service framework for low power playback devices
CN116489177A (en) IO access method and device based on block storage, electronic equipment and medium
CN116244231A (en) Data transmission method, device and system, electronic equipment and storage medium
CN115623057A (en) RDMA (remote direct memory Access) -based connection establishing method, device, equipment and storage medium
CN112243258B (en) Method and device for determining user perception rate
CN113055350B (en) Data transmission method, device, equipment and readable storage medium
CN110391952B (en) Performance analysis method, device and equipment
Marian et al. Analysis of Different SaaS Architectures from a Trust Service Provider Perspective
WO2019157062A1 (en) System and method for decentralized data storage
WO2023213101A1 (en) Communication method, apparatus and device
CN111221858B (en) Data processing method, device and equipment
US20230188341A1 (en) Cryptographic operations in edge computing networks

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 40057915

Country of ref document: HK

TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20240306

Address after: # 03-06, Lai Zan Da Building 1, 51 Belarusian Road, Singapore

Applicant after: Alibaba Innovation Co.

Country or region after: Singapore

Address before: Room 01, 45th Floor, AXA Building, 8 Shanton Road, Singapore

Applicant before: Alibaba Singapore Holdings Ltd.

Country or region before: Singapore