CN112433862A - Data aggregation implementation system and equipment - Google Patents
Data aggregation implementation system and equipment Download PDFInfo
- Publication number
- CN112433862A CN112433862A CN202011270367.7A CN202011270367A CN112433862A CN 112433862 A CN112433862 A CN 112433862A CN 202011270367 A CN202011270367 A CN 202011270367A CN 112433862 A CN112433862 A CN 112433862A
- Authority
- CN
- China
- Prior art keywords
- data
- interface
- aggregator
- splicing
- send
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 230000002776 aggregation Effects 0.000 title claims abstract description 21
- 238000004220 aggregation Methods 0.000 title claims abstract description 21
- 230000004044 response Effects 0.000 claims abstract description 7
- 238000000034 method Methods 0.000 claims description 23
- 230000008569 process Effects 0.000 claims description 20
- 238000013507 mapping Methods 0.000 claims description 3
- 230000006978 adaptation Effects 0.000 abstract description 6
- 238000006243 chemical reaction Methods 0.000 abstract description 4
- 238000004806 packaging method and process Methods 0.000 abstract 1
- 238000011161 development Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- VYZAMTAEIAYCRO-UHFFFAOYSA-N Chromium Chemical compound [Cr] VYZAMTAEIAYCRO-UHFFFAOYSA-N 0.000 description 2
- 235000014510 cooky Nutrition 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004904 shortening Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
Abstract
The invention provides a data aggregation realization system and a device, wherein the system comprises a data aggregator, and the data aggregator comprises: the interface request tool is used for requesting corresponding multiple groups of required data from multiple back-end interfaces through parallel requests and/or serial requests; the data processor is used for receiving a plurality of groups of required data from the interface request tool, removing redundant fields of the required data and splicing the required data to obtain spliced data; and the data output module is used for packaging the splicing data, the corresponding state codes and the interface information of each back-end interface, sending the splicing data to the front end in a fixed format, and sending the splicing data to the front end in response to receiving the exception codes sent by the interface request tool and/or the data processor after exception occurs. The invention can lead the front-end developer to flexibly carry out data conversion and aggregation on a plurality of back-end interfaces according to the requirements of different terminals, solves the problem that the back end needs to carry out interface adaptation according to the requirements of the front end, and realizes the decoupling of the front-end service data and the back-end service data.
Description
Technical Field
The invention relates to the technical field of software development, in particular to a system and equipment for realizing data aggregation.
Background
In the era of front-back separation, "one cloud with multiple ends" has become a development trend, and the front end is not limited to a PC end page, a mobile phone end, an applet end and other terminals, and is also more and more important. However, different terminals have different requirements for data and interfaces: if the mobile terminal is to save network overhead and accelerate page loading speed, the number of interface requests should be reduced as much as possible, and redundant fields should be removed; the PC side browser has better performance, the most abundant page and more fields needed by the interface. However, at present, when the back-end micro service is in the way, if interface adaptation is performed for different terminals, the back-end service and the front-end UI are seriously coupled, and the development efficiency is reduced.
One commonly used scheme at present is to use a backend gateway to perform data conversion and aggregation, and does not need to perform adaptation of different ends in a specific service interface, which solves the above problems to a certain extent, but still needs to pay attention to a front-end UI in a backend development process, and does not realize complete decoupling of the front end and the backend, thereby reducing development efficiency.
Disclosure of Invention
In view of this, the present invention is to provide a data aggregation implementation system, a storage medium, and a device, so as to solve the problem in the prior art that a backend developer needs to perform multiple adaptations on a backend interface.
Based on the above object, the present invention provides a data aggregation implementation system, which includes a data aggregator, where the data aggregator includes:
the interface request tool is configured to request a plurality of corresponding groups of required data from a plurality of backend interfaces through parallel requests and/or serial requests;
the data processor is configured to receive a plurality of groups of required data from the interface request tool, remove redundant fields of the required data and splice the required data to obtain spliced data; and
and the data output module is configured to pack the splicing data, the corresponding state codes thereof and the interface information of each back-end interface, send the splicing data and the corresponding state codes to the front end in a fixed format, and send the splicing data and the corresponding state codes to the front end in response to receiving the exception codes sent by the interface request tool and/or the data processor after exception occurs.
In some embodiments, the parallel requests are configured for mutually independent required data; the serial request is configured for interdependent desired data.
In some embodiments, the data aggregator further comprises a mapping unit configured to store all backend interfaces and their mapped input fields, and to accept the data aggregator to traverse all backend interfaces according to the input fields and find multiple backend interfaces.
In some embodiments, the system further comprises a process manager configured to open the plurality of data aggregators.
In some embodiments, the process manager is further configured to restart the data aggregator in response to a worker process exception thereto.
In some embodiments, the system further comprises a log management module configured to record a working log of the data aggregator and the process manager.
In some embodiments, the system further includes an alarm module configured to send alarm information of its corresponding level according to the logs of different levels reported by the log management module.
In some embodiments, the alarm module is further configured to send the alarm information of the corresponding type according to the different types of logs reported by the log management module.
In some embodiments, the fixed format is a JSON format.
In another aspect of the invention, there is also provided a computer apparatus comprising any of the systems as described above.
The invention has at least the following beneficial technical effects:
according to the invention, through the data aggregator, a front-end developer can flexibly perform data conversion and aggregation on a plurality of back-end interfaces according to the data requirements of different terminals, the problem that the back end needs to perform interface adaptation according to the front-end UI requirements is solved, the complete decoupling of the front-end UI and the back-end service data is realized, the development efficiency is greatly improved, and the iteration speed of a product is improved; and the front end can optimize and combine the rear-end interface request, thereby improving the request efficiency, shortening the request time and improving the performance of the front end.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a schematic diagram of an embodiment of a data aggregation implementation system provided in accordance with the present invention;
fig. 2 is a schematic hardware structure diagram of an embodiment of a computer device implementing the data aggregation implementation system according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two non-identical entities with the same name or different parameters, and it is understood that "first" and "second" are only used for convenience of expression and should not be construed as limiting the embodiments of the present invention. Furthermore, the terms "comprises" and "comprising," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements does not include all of the other steps or elements inherent in the list.
In view of the above object, a first aspect of the embodiments of the present invention provides an embodiment of a data aggregation implementation system. Fig. 1 is a schematic diagram illustrating an embodiment of a data aggregation implementation system provided by the present invention. As shown in fig. 1, the data aggregation implementation system of the present invention includes a data aggregator 20, where the data aggregator 20 includes: an interface request tool 201 configured to request corresponding sets of required data from a plurality of backend interfaces 10 by parallel requests and/or serial requests; a data processor 202 configured to receive multiple sets of required data from the interface request tool, remove redundant fields from the data, and perform splicing to obtain spliced data; and a data output module 203, configured to package the splicing data and the corresponding status code thereof and the interface information of each backend interface 10 and send the packaged splicing data and the corresponding status code to the front end in JSON format, and send the splicing data and the interface information to the front end 30 in response to receiving an exception code sent by the interface request tool 201 and/or the data processor 202 after an exception occurs.
According to the embodiment of the invention, through the data aggregator 20, a front-end developer can flexibly perform data conversion and aggregation on a plurality of rear-end interfaces 10 according to data requirements of different terminals, the problem that the rear end needs to perform interface adaptation according to the requirements of a front-end UI is solved, complete decoupling of the front-end UI and rear-end service data is realized, the development efficiency is greatly improved, and the iteration speed of a product is improved; and the front end can optimize and combine the requests of the rear-end interface 10, thereby improving the request efficiency, shortening the request time and improving the performance of the front end. Preferably, the data aggregation implementation system of this embodiment is implemented by node.js, which is a JavaScript operating environment based on Chrome V8 engine; the JavaScript is a lightweight and timely compiling programming language with function priority; chrome V8 is a high-performance JavaScript interpreter; the data aggregation implementation system has the characteristics of single thread, asynchronous input and output and high concurrency support.
In some embodiments, the parallel requests are configured for mutually independent required data; the serial request is configured for interdependent desired data. In this embodiment, the interface request tool 201 is specifically a request tool of an API interface (application program interface). The API interface request tool can use a data request module (such as an axios module) to request the required data of the back end, parallel requests are used through the independent required data, serial requests are used through the dependent required data, the request efficiency can be improved, front end request parameters are sorted aiming at different back end interfaces 10, header head information such as cookies and tokens is forwarded, and if the back end interfaces 10 make mistakes, retry operation is carried out; the cookie is data stored on the local terminal of the user and can be automatically carried when interacting with the back end; token represents a token for computer system user authentication.
In some embodiments, the data aggregator 20 further comprises a mapping unit configured to store all backend interfaces 10 and their mapped input fields, and to accept the data aggregator 20 to traverse all backend interfaces 10 according to the input fields and find a plurality of backend interfaces 10.
In some embodiments, the system further comprises a process manager configured to start the plurality of data aggregators 20. By starting multiple data aggregator 20 instances, using a cluster mode, and load balancing through load balancing, load balancing on each instance can ensure high concurrency and high availability of the data aggregator 20, and ensure normal operation of the data aggregator 20 in the event of a single instance error or a large number of requests. By setting weights for different working processes, the multi-process load balance under the cluster mode is realized, so that the high concurrency processing capability of the system is improved.
In some embodiments, the process manager is further configured to restart the data aggregator 20 in response to a work process exception thereto. Through the operation, the work process can be guarded, and the usability of the work process is ensured.
In some embodiments, the system further includes a log management module configured to record the working logs of the data aggregator 20 and the process manager. The log management module can record input fields, a requested rear-end interface, required data, redundant fields, splicing data and the like, and find the source and the processing process of the data through logs, so that errors are easier to find.
In some embodiments, the system further includes an alarm module configured to send alarm information of its corresponding level according to the logs of different levels reported by the log management module. In this embodiment, the alarm information may be sent to a mailbox of the developer to remind the developer to process in time.
In some embodiments, the alarm module is further configured to send the alarm information of the corresponding type according to the different types of logs reported by the log management module. In the embodiment, data tracing and error processing can be facilitated by reporting an error to the related log of the data aggregator; and recording an error log after the related log of the process manager is reported, and immediately sending an email of the alarm information.
In some embodiments, the fixed format is a JSON format. Json (javascript Object notification) is a lightweight data exchange format that stores and represents data in a text format completely independent of programming languages, based on a subset of ECMAScript (js specification set by the european computer association); easy to read and write by human and easy to analyze and generate by machine.
In a second aspect of the embodiments of the present invention, there is also provided a computer device including the system according to any one of the above-mentioned embodiments.
Fig. 2 is a schematic hardware structure diagram of an embodiment of a computer device for implementing a data aggregation implementation system according to the present invention. Taking the computer device shown in fig. 2 as an example, the computer device includes a processor 301 and a memory 302, and may further include: an input device 303 and an output device 304. The processor 301, the memory 302, the input device 303 and the output device 304 may be connected by a bus or other means, and fig. 2 illustrates the connection by a bus as an example. The input device 303 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the data aggregation implementation system. The output means 304 may comprise a display device such as a display screen. The processor 301 executes various functional applications of the server and data processing by executing nonvolatile software programs, instructions, and modules stored in the memory 302 to implement the data aggregation implementation system of the above-described embodiment.
Finally, it should be noted that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.
The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with the following components designed to perform the functions herein: a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and/or any other such configuration.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items. The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A data aggregation implementation system, comprising a data aggregator, the data aggregator comprising:
the interface request tool is configured to request a plurality of corresponding groups of required data from a plurality of backend interfaces through parallel requests and/or serial requests;
the data processor is configured to receive the multiple groups of required data from the interface request tool, remove redundant fields of the multiple groups of required data and splice the multiple groups of required data to obtain spliced data; and
and the data output module is configured to pack the splicing data, the corresponding state codes thereof and the interface information of each back-end interface, send the splicing data and the corresponding state codes to the front end in a fixed format, and send the splicing data and the corresponding state codes to the front end in response to receiving the exception codes sent by the interface request tool and/or the data processor after exception occurs.
2. The system of claim 1, wherein the parallel requests are configured for mutually independent desired data; the serial request is configured for interdependent desired data.
3. The system of claim 1, wherein the data aggregator further comprises a mapping unit configured to store all backend interfaces and their mapped input fields, and to accept the data aggregator traversing all backend interfaces according to input fields and finding the plurality of backend interfaces.
4. The system of claim 1, further comprising a process manager configured to open a plurality of the data aggregators.
5. The system of claim 4, wherein the process manager is further configured to restart the data aggregator in response to a worker process exception to the data aggregator.
6. The system of claim 4, further comprising a log management module configured to record a working log of the data aggregator and the process manager.
7. The system according to claim 6, further comprising an alarm module configured to send alarm information of corresponding levels according to different levels of logs reported by the log management module.
8. The system of claim 7, wherein the alarm module is further configured to send the alarm information of the corresponding type according to the different types of logs reported by the log management module.
9. The system of claim 1, wherein the fixed format is a JSON format.
10. A computer device comprising a system as claimed in any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011270367.7A CN112433862A (en) | 2020-11-13 | 2020-11-13 | Data aggregation implementation system and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011270367.7A CN112433862A (en) | 2020-11-13 | 2020-11-13 | Data aggregation implementation system and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112433862A true CN112433862A (en) | 2021-03-02 |
Family
ID=74701276
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011270367.7A Withdrawn CN112433862A (en) | 2020-11-13 | 2020-11-13 | Data aggregation implementation system and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112433862A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114338282A (en) * | 2021-11-16 | 2022-04-12 | 飞诺门阵(北京)科技有限公司 | Security gateway and data processing method thereof |
-
2020
- 2020-11-13 CN CN202011270367.7A patent/CN112433862A/en not_active Withdrawn
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114338282A (en) * | 2021-11-16 | 2022-04-12 | 飞诺门阵(北京)科技有限公司 | Security gateway and data processing method thereof |
| CN114338282B (en) * | 2021-11-16 | 2024-01-26 | 飞诺门阵(北京)科技有限公司 | Security gateway and data processing method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11005969B2 (en) | Problem solving in a message queuing system in a computer network | |
| CN107729139B (en) | Method and device for concurrently acquiring resources | |
| EP3082295B1 (en) | Fault management apparatus, device and method for network function virtualization (nfv) | |
| CN111581948B (en) | Document analysis method, device, equipment and storage medium | |
| US20080010497A1 (en) | Selecting a Logging Method via Metadata | |
| US9497095B2 (en) | Dynamic control over tracing of messages received by a message broker | |
| US20200159622A1 (en) | Rule based failure addressing | |
| US10592314B2 (en) | Message queueing in middleware by a message broker | |
| EP3933591A1 (en) | Stream computing method and apparatus based on dag interaction | |
| US20210406102A1 (en) | Method and apparatus for providing asynchronicity to microservice application programming interfaces | |
| CN107977260B (en) | Task submitting method and device | |
| CN112433862A (en) | Data aggregation implementation system and equipment | |
| US11645109B2 (en) | Managing failures in edge computing environments | |
| US11449407B2 (en) | System and method for monitoring computing platform parameters and dynamically generating and deploying monitoring packages | |
| CN109144676A (en) | A kind of self-starting detection method, device and the server of application program | |
| CN110362294A (en) | Development task executes method, apparatus, electronic equipment and storage medium | |
| US20210026610A1 (en) | Method, device and terminal apparatus for code execution and computer readable storage medium | |
| CN111698109A (en) | Method and device for monitoring log | |
| US11210156B1 (en) | Intelligent distributed tracing | |
| US20220201091A1 (en) | Coordinating requests actioned at a scalable application | |
| US11061743B2 (en) | Event loop optimization through event ordering | |
| CN111813621A (en) | Data processing method, device, equipment and medium based on Flume data middlebox | |
| CN116992065B (en) | Graph database data importing method, system, electronic equipment and medium | |
| US20230028407A1 (en) | In-band modification of event notification preferences for server events | |
| US8661296B2 (en) | Dynamic data store for failed jobs in a systems complex |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210302 |
|
| WW01 | Invention patent application withdrawn after publication |