CN115952050A - Reporting method and device for organization service buried point data - Google Patents
Reporting method and device for organization service buried point data Download PDFInfo
- Publication number
- CN115952050A CN115952050A CN202211589292.8A CN202211589292A CN115952050A CN 115952050 A CN115952050 A CN 115952050A CN 202211589292 A CN202211589292 A CN 202211589292A CN 115952050 A CN115952050 A CN 115952050A
- Authority
- CN
- China
- Prior art keywords
- data
- buried point
- reporting
- point data
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a reporting method and device of mechanism service buried point data, and relates to the technical field of smart cities and smart medical treatment. One embodiment of the method comprises: receiving buried point data reported by a mechanism service, and asynchronously serializing and storing the buried point data to a local disk; packaging the buried point data according to a preset data packaging rule to generate a data packet, and generating a data reporting request according to the data packet; and according to a preset data reporting strategy, reporting the buried point data by submitting the data reporting request. According to the embodiment, the embedded data is written into a local disk file of the service, data packaging is completed according to configuration asynchronous batch compression, and then the embedded data is reported in a client request mode such as an HTTP request or an SOAP request, so that the bandwidth is saved, normal operation of the mechanism service is protected under a scene that the embedded data is frequently generated, and the stability and the throughput capacity of the mechanism service are improved.
Description
Technical Field
The invention relates to the technical field of smart cities and smart medical treatment, in particular to a reporting method and device of mechanism service buried point data.
Background
In smart city construction, in order to centrally manage the operating states of services of organizations (such as hospitals, governments, and the like) in a city, it is necessary to collect heartbeat information and page access data of the services of the organizations, monitor and manage data interaction between the services of the organizations, and the like. Currently, when performing centralized management on each organization service, data acquisition and reporting are often completed by sending a HTTP (Hyper Text Transfer Protocol) request or a SOAP (Simple Object Access Protocol) request to each organization service to obtain buried point data.
However, many facilities in a city, for example: administrative units such as hospitals and governments mostly walk private network lines, and bandwidth resources which can be provided for building interconnected smart cities are very limited. Regardless of whether the data of the embedded point is obtained through the HTTP request or the SOAP request, each mechanism needs to pass through a plurality of devices such as a gateway and a board-hopping machine, so that the response time of the request is obviously higher than that of a common service, a large amount of bandwidth resources are occupied, the throughput performance of services of each mechanism is also deteriorated, and if the data of the embedded point is frequently reported, the normal service is finally in an unavailable state, and serious consequences are brought.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for reporting mechanism service buried point data, which can write the buried point data into a local disk file of a service, complete data packing according to configuration asynchronous batch compression, and then report the buried point data by using client requests such as HTTP requests or SOAP requests, so that bandwidth is saved, normal operation of the mechanism service is protected in a scenario where the buried point data is frequently generated, and stability and throughput of the mechanism service are improved.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a reporting method of mechanism service buried point data, including:
receiving buried point data reported by a mechanism service, and asynchronously serializing and storing the buried point data to a local disk;
packaging the buried point data according to a preset data packaging rule to generate a data packet, and generating a data reporting request according to the data packet;
and according to a preset data reporting strategy, reporting the buried point data by submitting the data reporting request.
Optionally, asynchronously serializing the buried point data to be stored in a local disk, including: and asynchronously printing the buried point data to an independent file of a local disk in a log printing mode so as to asynchronously store the buried point data in series to the local disk.
Optionally, the data packing rule includes a data packing size and a data packing manner; packing the buried point data according to a preset data packing rule to generate a data package, wherein the data package comprises the following steps: segmenting the buried point data according to the data packing size to obtain at least one data body meeting the data packing size, wherein different data bodies are stored in a line-changing mode; and packaging each data body according to the data packaging mode to generate a data packet.
Optionally, the data reporting policy includes a request frequency and a reported data size; according to a preset data reporting strategy, the data reporting method for reporting buried point data by submitting the data reporting request comprises the following steps: calculating the reporting frequency of the buried point data according to the request frequency and the size of the reported data; and according to the reporting frequency, reporting the buried point data by submitting the data reporting request.
Optionally, submitting the data reporting request to report the buried point data includes: submitting the data reporting request to a thread pool, processing data of a data packet in the data reporting request through the thread pool, and performing index calculation according to the processed data to finish data reporting of the buried point.
Optionally, the buried point data has a type, the data packet in the data reporting 5 request is processed through the thread pool, and index calculation is performed according to the processed data,
the method comprises the following steps: performing data processing on the data packet in the data reporting request through the thread pool, so as to perform data classification according to the type of buried point data, and performing index calculation according to the classified data; the method further comprises the following steps: and after the buried point data is reported, responding to a buried point data query request, acquiring a type corresponding to the buried point data to be queried, and querying the buried point data 0 according to the type.
According to another aspect of the embodiments of the present invention, there is provided a reporting apparatus for mechanism service buried point data, including:
the data storage module is used for receiving buried point data reported by the organization service and asynchronously serializing and storing the buried 5-point data to a local disk;
the data packing module is used for packing the buried point data according to a preset data packing rule to generate a data packet and generating a data reporting request according to the data packet;
and the data reporting module is used for reporting the buried point data by submitting the data reporting request according to a preset data reporting strategy.
Optionally, the data packing rule includes a data packing size and a data packing manner;
the data packing module is further configured to: and partitioning the buried point data according to the data packing size to obtain at least one data volume satisfying the data packing size,
performing line feed storage on different data volumes; and packaging each data body according to the data packaging method 5 to generate a data packet.
According to still another aspect of an embodiment of the present invention, there is provided an electronic apparatus including: one or more processors; and the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors 0 realize the reporting method of the agency service buried point data provided by the embodiment of the invention.
According to still another aspect of the embodiments of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements the method for reporting agency service buried point data provided by the embodiments of the present invention.
One embodiment of the above invention has the following advantages or benefits: the method comprises the steps of receiving buried point data reported by a mechanism service, and asynchronously serializing and storing the buried point data to a local disk; packaging the buried point data according to a preset data packaging rule to generate a data packet, and generating a data reporting request according to the data packet; according to a preset data reporting strategy, by the technical scheme of submitting a data reporting request to report the buried point data, the buried point data can be written into a local disk file of a service, data packaging is completed according to configuration asynchronous batch compression, and then the buried point data is reported in a client request mode such as an HTTP (hyper text transport protocol) request or SOAP (simple object access protocol) request mode, so that the bandwidth is saved, normal operation of mechanism services is protected under a scene that the buried point data is frequently generated, and the stability and throughput capacity of the mechanism services are improved.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram illustrating main steps of a reporting method of organization service buried point data according to an embodiment of the present invention;
fig. 2 is a schematic diagram illustrating a data reporting implementation principle according to an embodiment of the present invention;
FIG. 3 is a schematic block diagram of an apparatus for reporting organization service buried point data according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It should be noted that, in the technical solution disclosed in the present invention, the aspects of collecting, updating, analyzing, processing, using, transmitting, storing and the like of the personal information of the related user all conform to the regulations of the related laws and regulations, and are used for legal purposes without violating the good customs of the public order. Necessary measures are taken for the personal information of the user, the illegal access to the personal information data of the user is prevented, and the personal information safety, the network safety and the national safety of the user are maintained.
In a traditional embedded data processing mode, when embedded data is generated, an HTTP request is sent by a mechanism, and a receiving service receives the data and then processes the data, but the point-to-point mode has great challenges on the throughput and stability of a system and has high data retry realization cost. In order to solve the technical problem, the invention provides a reporting method of mechanism service buried point data, which writes the buried point data into a local disk file of a service, completes data packing according to configuration asynchronous batch compression, and then reports the data by adopting a client request mode such as an HTTP request or an SOAP request. Therefore, the bandwidth is saved, normal operation of the mechanism service is protected under the scene that the buried point data is frequently generated, and the stability and the throughput capacity of the mechanism service are improved.
Fig. 1 is a schematic diagram illustrating main steps of a reporting method of organization service buried point data according to an embodiment of the present invention. As shown in fig. 1, the reporting method of the agency service buried point data according to the embodiment of the present invention mainly includes the following steps S101 to S103.
Step S101: and receiving buried point data reported by the service of the mechanism, and asynchronously serializing and storing the buried point data to a local disk. In the embodiment of the invention, a Software Development Kit (SDK) package for reporting buried point data is installed and deployed in a private cloud of independent deployment of mechanism services, and after the buried point data is collected, each mechanism service can process and report the buried point data by calling the SDK package. After the buried point data is collected, firstly, an interface of the SDK packet is called to transmit the buried point data to be reported to the SDK packet, so that the buried point data is asynchronously serialized and stored in a local disk of the organization service. In an embodiment of the present invention, the SDK package is, for example, a compressed package file that can be integrated into the data processing framework spring boot.
According to an embodiment of the present invention, when asynchronously serializing and storing the buried point data to the local disk, the method may specifically include: and asynchronously printing the buried point data into an independent file of the local disk in a log printing manner so as to asynchronously serialize and store the buried point data into the local disk. Specifically, the method for asynchronously printing the embedded data to the log in the independent file of the local disk can be realized through the log collection device logstack. In the embodiment of the invention, the data of the buried point is not uploaded immediately after being collected, but is collected and stored in an asynchronous mode and then packaged and uploaded, so that the bandwidth is saved and the stability of the organization service is ensured.
Step S102: and packaging the buried point data according to a preset data packaging rule to generate a data packet, and generating a data reporting request according to the data packet. In the embodiment of the invention, after storing the received buried point data of the organization service in the local disk, the SDK package can asynchronously package the buried point data stored in the local disk in batch to form a data package, and generate a data reporting request according to the data package to request for buried point data reporting, where the data reporting request is, for example, an HTTP request or a SOAP request.
According to one embodiment of the invention, the data packing rule comprises a data packing size and a data packing manner. In addition, according to the needs of a specific implementation scenario, the data packing rule may further include a cut-amount configuration, a limited policy configuration, and the like. The slicing configuration is used for slicing the large-flow data, and asynchronous transmission is performed, so that stability of a Central Processing Unit (CPU) and bandwidth of the system is guaranteed. Packing the buried point data according to a preset data packing rule to generate a data packet, which may specifically include: segmenting the buried point data according to the data packing size to obtain at least one data body meeting the data packing size, wherein different data bodies are stored in a line-changing mode; and packaging each data body according to the data packaging mode to generate a data package. Specifically, at least one piece of buried point data that does not exceed the size of the data packet may be divided into one data volume according to a preset data packet size (which is a limit), different data volumes may be stored in a line feed manner, and each data volume may be packed according to a data packet mode to generate a data packet. The data packing mode can be designed independently aiming at different types of buried point data, can also be a unified data packing mode, and can also be used for compressing data to reduce the data volume through the data packing mode, so that the bandwidth is saved, and the stability of mechanism service is also ensured. The data volume may be packed and compressed by a ZSTD (Zstandard) packing system, for example. The compression ratio and compression/decompression performance of the ZSTD are outstanding. The support of each system is relatively good, for example: the Liunx kernel, the HTTP protocol, and some big data tools, etc.
Step S103: and according to a preset data reporting strategy, reporting the buried point data by submitting the data reporting request. In the embodiment of the present invention, the data reporting policy includes, for example, a size of data to be reported each time, a request frequency, a reporting priority policy (for example, a priority for weight configuration, a priority for interface reporting, a priority for time, a priority for providing an interface for external use, and the like). And reporting the data of the buried points according to the data reporting strategy, finishing the reporting of the data of the buried points according to the flow configuration, and trying to resubmit a data reporting request under the condition that the reporting is not successful. In addition, because the bandwidth resources of the mechanism are limited, the configuration execution message discarding strategy can be started on the premise of ensuring the stability and the availability of the service; or the flow reported by the system reaches the upper limit, and a message discarding strategy is also triggered. And after the system parameter adjustment is recovered, closing the message discarding strategy, and reporting the message normally by the system. For the discarded message, after the system is recovered, the message data can be retrieved, because the data is stored in the local disk in a serialized manner, a file is formed.
According to one embodiment of the present invention, the data reporting policy includes, for example, a request frequency and a reported data size. According to a preset data reporting strategy, the data reporting request is submitted to report the buried point data, which specifically includes: calculating the reporting frequency of the buried point data according to the request frequency and the size of the reported data; and according to the reporting frequency, reporting the buried point data by submitting the data reporting request. Because bandwidth resources of each access mechanism side of the smart city are limited, a current request amount needs to be calculated according to configuration, and a current buried point data reporting frequency is generated, for example: the request frequency is configured in advance to be 5 times/second, and the data volume reported once is not more than 500KB, then according to the configuration, the data reporting frequency can be calculated to be 10 times/second, and each time is 400KB. The specific data needs to be observed and adjusted according to the actual scene of the interface, so that sufficient margin is ensured, and the stability of the system and the throughput of the service are ensured.
According to one embodiment of the present invention, submitting the data reporting request to report the buried point data may specifically include: submitting the data reporting request to a thread pool, processing data of a data packet in the data reporting request through the thread pool, and performing index calculation according to the processed data to finish data reporting of the buried point. After the data reporting request is submitted to the thread pool, the threads in the thread pool can perform data processing on the data packet in the data reporting request to complete the receiving and processing of the data, and the processed data is sent to Kafka in a message form. After receiving the data, kafka completes index calculation through a Flink big data calculation engine, or directly performs inventory dropping storage to complete data reporting of buried points.
According to an embodiment of the invention, the buried point data has a type, for example: user click events, browsing events, execution logs of the system, request logs of the interface, response messages requesting third party systems, and the like in the buried point data. By analyzing and storing the buried point data according to types, subsequent management, analysis, use and the like can be facilitated. According to the embodiment of the present invention, the data processing is performed on the data packet in the data reporting request through the thread pool, and the index calculation is performed according to the processed data, which may specifically include: and performing data processing on the data packet in the data reporting request through the thread pool, so as to perform data classification according to the type of the buried point data, and performing index calculation according to the classified data. In addition, in the method of the present invention, the buried point data may be searched or displayed according to the type. Specifically, after the buried point data is reported, responding to a buried point data query request, obtaining a type corresponding to the buried point data to be queried, and querying the buried point data according to the type. When the data needs to be analyzed or the data of the buried point list needs to be retrieved and inquired, the data can be obtained through an API (application program interface) of the ClickHouse of the data warehouse. For some index data, the index result can be pushed to the Mysql or Redis database for storage through a big data platform, so that the index data can be conveniently inquired according to service scenes and requirements.
According to the embodiment of the invention, the acquisition and reporting of the buried point data are carried out in an asynchronous mode, so that the bandwidth is saved, and the stability of the organization service is ensured; the data reporting request processing is carried out by adopting a multithreading mode in the thread pool, the throughput capacity of the data of the embedded point is greatly improved, and meanwhile, the configuration management function and the configuration real-time issuing function of a mechanism are achieved, so that the unified management of multiple mechanisms is ensured; the method is characterized in that a ClickHouse and a data warehouse are used for storing mass data, and meanwhile, each mechanism is operated, a data query management interface is provided, and the closed loop of data management of the whole buried point is completed.
Fig. 2 is a schematic diagram illustrating a data reporting implementation principle according to an embodiment of the present invention. As shown in fig. 2, the implementation principle of data reporting in the embodiment of the present invention is described by taking reporting of service buried point data of an organization a and an organization B as an example. After the mechanism A service and the mechanism B service acquire the buried point data, the buried point data reporting SDK deployed on the private cloud of each mechanism A service and the buried point data reporting SDK deployed on the private cloud of each mechanism B service are called to write the buried point data of the mechanism A service and the buried point data of the mechanism B service into the local disk of the mechanism A service and the local disk of the mechanism B service respectively. And then, summarizing the buried point data to be uploaded in the local disk served by the mechanism A and the local disk served by the mechanism B through a file collecting tool, then carrying out uniform batch packing and compression processing on the buried point data according to a configured data packing rule, and generating a data reporting request. And then, reporting the buried point data to a data receiving service by submitting a data reporting request according to the configured data reporting strategy. If the data reporting request cannot be successfully submitted, the data reporting request is tried to be submitted again so as to ensure the integrity and accuracy of data reporting. In addition, because the bandwidth resource of the mechanism is limited, on the premise of ensuring the stability and the availability of the service, the configuration execution message discarding strategy is started; or the flow reported by the system reaches the upper limit, and a message discarding strategy is also triggered. And after the system parameters are adjusted and recovered, closing the message discarding strategy, and reporting the message normally by the system. For the discarded message, after the system is recovered, the message data can be retrieved, because the data is stored in the local disk in a serialized manner, a file is formed.
Fig. 3 is a schematic block diagram of a reporting apparatus of agency service buried point data according to an embodiment of the present invention. As shown in fig. 3, a reporting apparatus 300 for agency service site data according to an embodiment of the present invention mainly includes: a data storage module 301, a data packing module 302 and a data reporting module 303.
The data storage module 301 is configured to receive buried point data reported by a mechanism service, and asynchronously serialize and store the buried point data to a local disk;
a data packing module 302, configured to pack the buried point data according to a preset data packing rule to generate a data packet, and generate a data reporting request according to the data packet;
and a data reporting module 303, configured to submit the data reporting request according to a preset data reporting policy to perform data reporting of the buried point.
According to an embodiment of the invention, the data storage module 301 may further be configured to: and asynchronously printing the buried point data to an independent file of a local disk in a log printing mode so as to asynchronously store the buried point data in series to the local disk.
According to another embodiment of the present invention, the data packing rule includes a data packing size and a data packing manner; the data packing module 302 may also be configured to: segmenting the buried point data according to the data packing size to obtain at least one data body meeting the data packing size, wherein different data bodies are stored in a line-changing mode; and packaging each data body according to the data packaging mode to generate a data packet.
According to another embodiment of the present invention, the data reporting policy includes a request frequency and a reported data size; the data reporting module 303 may be further configured to: calculating the reporting frequency of the buried point data according to the request frequency and the size of the reported data; and according to the reporting frequency, reporting the buried point data by submitting the data reporting request.
According to another embodiment of the present invention, the data reporting module 303 may further be configured to: submitting the data reporting request to a thread pool, processing data of a data packet in the data reporting request through the thread pool, and performing index calculation according to the processed data to finish data reporting of the buried point.
According to another embodiment of the present invention, the buried point data has a type, and when the data reporting module 303 performs data processing on the data packet in the data reporting request through the thread pool and performs index calculation according to the processed data, the data reporting module may further be configured to: performing data processing on the data packet in the data reporting request through the thread pool, so as to perform data classification according to the type of buried point data, and performing index calculation according to the classified data; moreover, the reporting apparatus 300 for the agency service buried point data of the present invention may further include a data query module (not shown in the figure), configured to: and after the report of the buried point data is finished, responding to a buried point data query request, acquiring a type corresponding to the buried point data to be queried, and querying the buried point data according to the type.
According to the technical scheme of the embodiment of the invention, the buried point data reported by the service of a receiving mechanism is received, and the buried point data is asynchronously stored to a local disk in a serialized manner; the method comprises the steps that buried point data are packed according to a preset data packing rule to generate a data packet, and a data reporting request is generated according to the data packet; according to a preset data reporting strategy, by the technical scheme of submitting a data reporting request to report the buried point data, the buried point data can be written into a local disk file of a service, data packaging is completed according to configuration asynchronous batch compression, and then the buried point data is reported in a client request mode such as an HTTP (hyper text transport protocol) request or SOAP (simple object access protocol) request mode, so that the bandwidth is saved, normal operation of mechanism services is protected under a scene that the buried point data is frequently generated, and the stability and throughput capacity of the mechanism services are improved.
Fig. 4 shows an exemplary system architecture 400 to which the reporting method of the agency service buried point data or the reporting apparatus of the agency service buried point data according to the embodiment of the present invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The terminal devices 401, 402, 403 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 405 may be a server providing various services, such as a background management server (for example only) providing support for a data reporting website browsed by a user using the terminal devices 401, 402, and 403. The background management server can receive the buried point data reported by the organization service for the received data such as the buried point data reporting request and the like, and asynchronously serializes and stores the buried point data to a local disk; packaging the buried point data according to a preset data packaging rule to generate a data packet, and generating a data reporting request according to the data packet; and according to a preset data reporting strategy, processing such as buried point data reporting and the like is carried out by submitting the data reporting request, and a processing result (such as a buried point data reporting result-only an example) is fed back to the terminal equipment.
It should be noted that the reporting method of the agency service buried point data provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the reporting apparatus of the agency service buried point data is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, a block diagram of a computer system 500 suitable for use with a terminal device or server implementing an embodiment of the invention is shown. The terminal device or the server shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU) 501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. Input-
An output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like;
a 5 output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. A drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary 0.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program 5 comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.
0 it should be noted that the computer readable media shown in the present invention can be computer readable signal media or computer readable storage media or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires 5, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution 0 system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units or modules described in the embodiments of the present invention may be implemented by software, or may be implemented by hardware. The described units or modules may also be provided in a processor, and may be described as: a processor comprises a data storage module, a data packing module and a data reporting module. The names of these units or modules do not form a limitation on the units or modules themselves in some cases, for example, the data storage module may also be described as "a module for receiving buried point data reported by an agency service and asynchronously serializing and storing the buried point data to a local disk".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not assembled into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving buried point data reported by a mechanism service, and asynchronously serializing and storing the buried point data to a local disk; packaging the buried point data according to a preset data packaging rule to generate a data packet, and generating a data reporting request according to the data packet; and according to a preset data reporting strategy, reporting the buried point data by submitting the data reporting request.
According to the technical scheme of the embodiment of the invention, the buried point data reported by the service of a receiving mechanism is received, and the buried point data is asynchronously stored to a local disk in a serialized manner; the method comprises the steps that buried point data are packed according to a preset data packing rule to generate a data packet, and a data reporting request is generated according to the data packet; according to a preset data reporting strategy, by means of the technical scheme that a data reporting request is submitted to report buried data, the buried data can be written into a local disk file of a service, data packing is completed according to configuration asynchronous batch compression, and then the buried data is reported in a client request mode such as an HTTP (hyper text transport protocol) request or SOAP (simple object access protocol) request mode, so that bandwidth is saved, normal operation of mechanism services is protected under a scene that the buried data is frequently generated, and stability and throughput capacity of the mechanism services are improved.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A reporting method of mechanism service buried point data is characterized by comprising the following steps:
receiving buried point data reported by a mechanism service, and asynchronously serializing and storing the buried point data to a local disk;
packaging the buried point data according to a preset data packaging rule to generate a data packet, and generating a data reporting request according to the data packet;
and according to a preset data reporting strategy, reporting the buried point data by submitting the data reporting request.
2. The method of claim 1, wherein asynchronously serializing the buried point data for storage to a local disk comprises:
and asynchronously printing the buried point data to an independent file of a local disk in a log printing mode so as to asynchronously serialize and store the buried point data to the local disk.
3. The method of claim 1, wherein the data packing rules include a data packing size and a data packing manner;
packing the buried point data according to a preset data packing rule to generate a data packet, wherein the data packet comprises the following steps:
segmenting the buried point data according to the data packing size to obtain at least one data body meeting the data packing size; wherein, different data volumes are stored in a line feed manner;
and packaging each data body according to the data packaging mode to generate a data packet.
4. The method of claim 1, wherein the data reporting policy comprises a request frequency and a reporting data size;
according to a preset data reporting strategy, the data reporting method for reporting buried point data by submitting the data reporting request comprises the following steps:
calculating the reporting frequency of the buried point data according to the request frequency and the reporting data size;
and according to the reporting frequency, reporting the buried point data by submitting the data reporting request.
5. The method of claim 1, wherein submitting the data reporting request for buried point data reporting comprises:
submitting the data reporting request to a thread pool, processing data of a data packet in the data reporting request through the thread pool, and performing index calculation according to the processed data to finish data reporting of the buried point.
6. The method as claimed in claim 5, wherein the buried point data has a type, the data processing is performed on the data packet in the data reporting request through the thread pool, and the index calculation is performed according to the processed data, including:
performing data processing on the data packet in the data reporting request through the thread pool, classifying the data according to the type of the buried point data, and performing index calculation according to the classified data;
the method further comprises the following steps: and after the buried point data is reported, responding to a buried point data query request, acquiring a type corresponding to the buried point data to be queried, and querying the buried point data according to the type.
7. A mechanism service buried point data reporting device is characterized by comprising:
the data storage module is used for receiving buried point data reported by the organization service and asynchronously serializing and storing the buried point data to a local disk;
the data packing module is used for packing the buried point data according to a preset data packing rule to generate a data packet and generating a data reporting request according to the data packet;
and the data reporting module is used for reporting the buried point data by submitting the data reporting request according to a preset data reporting strategy.
8. The apparatus of claim 7, wherein the data packing rules include a data packing size and a data packing manner;
the data packing module is further configured to:
segmenting the buried point data according to the data packing size to obtain at least one data body meeting the data packing size, wherein different data bodies are stored in a line-changing mode;
and packaging each data body according to the data packaging mode to generate a data package.
9. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211589292.8A CN115952050A (en) | 2022-12-12 | 2022-12-12 | Reporting method and device for organization service buried point data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211589292.8A CN115952050A (en) | 2022-12-12 | 2022-12-12 | Reporting method and device for organization service buried point data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115952050A true CN115952050A (en) | 2023-04-11 |
Family
ID=87289955
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211589292.8A Pending CN115952050A (en) | 2022-12-12 | 2022-12-12 | Reporting method and device for organization service buried point data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115952050A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117591381A (en) * | 2024-01-18 | 2024-02-23 | 南京研利科技有限公司 | Data reporting method and device |
-
2022
- 2022-12-12 CN CN202211589292.8A patent/CN115952050A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117591381A (en) * | 2024-01-18 | 2024-02-23 | 南京研利科技有限公司 | Data reporting method and device |
| CN117591381B (en) * | 2024-01-18 | 2024-04-09 | 南京研利科技有限公司 | Data reporting method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110147398B (en) | Data processing method, device, medium and electronic equipment | |
| CN108874640B (en) | Cluster performance evaluation method and device | |
| US9104985B2 (en) | Processing system using metadata for administering a business transaction | |
| US20100005134A1 (en) | Systems and methods for recording changes to a data store and propagating changes to a client application | |
| US9704207B2 (en) | Administering medical digital images in a distributed medical digital image computing environment with medical image caching | |
| US20130060723A1 (en) | Method and system for a smart agent for information management with feed aggregation | |
| GB2495824A (en) | Managing the failover operations of a storage unit in a cluster of computers | |
| CN110928934A (en) | Data processing method and device for business analysis | |
| CN110928905B (en) | Data processing method and device | |
| US11416294B1 (en) | Task processing for management of data center resources | |
| CN110427304A (en) | O&M method, apparatus, electronic equipment and medium for banking system | |
| US10331484B2 (en) | Distributed data platform resource allocator | |
| US20120284390A1 (en) | Guaranteed response pattern | |
| CN115952050A (en) | Reporting method and device for organization service buried point data | |
| CN113190517B (en) | Data integration method and device, electronic equipment and computer readable medium | |
| CN107357526A (en) | For the method and apparatus of network data, server and storage medium | |
| US9679262B2 (en) | Image index routing | |
| CN113190558A (en) | Data processing method and system | |
| CN107729394A (en) | Data Mart management system and its application method based on Hadoop clusters | |
| EP4216076A1 (en) | Method and apparatus of processing an observation information, electronic device and storage medium | |
| CN111427899A (en) | Method, device, equipment and computer readable medium for storing file | |
| CN112688982B (en) | User request processing method and device | |
| CN114035861A (en) | Cluster configuration method and device, electronic equipment and computer readable medium | |
| CN113778818A (en) | Method, apparatus, device and computer readable medium for optimizing system | |
| CN113378346A (en) | Method and device for model simulation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |