CN117743323A - Data processing method and device, electronic equipment and nonvolatile storage medium - Google Patents
Data processing method and device, electronic equipment and nonvolatile storage medium Download PDFInfo
- Publication number
- CN117743323A CN117743323A CN202311612925.7A CN202311612925A CN117743323A CN 117743323 A CN117743323 A CN 117743323A CN 202311612925 A CN202311612925 A CN 202311612925A CN 117743323 A CN117743323 A CN 117743323A
- Authority
- CN
- China
- Prior art keywords
- data
- user
- defined function
- hbase
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 23
- 238000007405 data analysis Methods 0.000 claims abstract description 49
- 230000002776 aggregation Effects 0.000 claims abstract description 48
- 238000004220 aggregation Methods 0.000 claims abstract description 48
- 238000012545 processing Methods 0.000 claims abstract description 38
- 238000000034 method Methods 0.000 claims abstract description 30
- 230000006870 function Effects 0.000 claims description 116
- 238000004590 computer program Methods 0.000 claims description 6
- 238000012935 Averaging Methods 0.000 claims description 5
- 230000004931 aggregating effect Effects 0.000 claims description 2
- 230000003993 interaction Effects 0.000 claims 1
- 238000004458 analytical method Methods 0.000 abstract description 16
- 238000005516 engineering process Methods 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data processing method, a data processing device, electronic equipment and a nonvolatile storage medium. Wherein the method comprises the following steps: determining a user-defined function and target parameters corresponding to the user-defined function, wherein the target parameters comprise: data fields corresponding to data analysis requirements; acquiring target data corresponding to the data field in the HBase storage system according to a user-defined function, wherein the user-defined function is at least used for indicating a query condition for querying the target data corresponding to the data field in the HBase storage system; and carrying out aggregation operation on the target data according to the user-defined function to obtain a data analysis result. The method and the device solve the technical problem of poor data processing performance caused by the fact that the related technology needs to issue withdrawal data and state storage when the dynamic table of the FlinkSQL tense table is associated with the HBase dynamic table for aggregation analysis.
Description
Technical Field
The present invention relates to the field of large data stream data processing technologies, and in particular, to a data processing method, apparatus, electronic device, and nonvolatile storage medium.
Background
When the related technology is used for associating the HBase dynamic table with the FlinkSQL tense table, the method is realized through a User-defined table function (User-Defined Table Function, UDTF), and withdrawn data and state storage are required to be issued when aggregation analysis is carried out, so that the performance of processing the data is reduced, and the association of non-primary key fields is not supported.
In view of the above problems, no effective solution has been proposed at present.
Disclosure of Invention
The embodiment of the application provides a data processing method, a device, electronic equipment and a nonvolatile storage medium, which at least solve the technical problem of poor data processing performance caused by the fact that withdrawn data and state storage are required to be issued when a relevant technology performs aggregation analysis on a FlankSQL tense table associated HBase dynamic table.
According to an aspect of the embodiments of the present application, there is provided a data processing method, including: determining a user-defined function and target parameters corresponding to the user-defined function, wherein the target parameters comprise: data fields corresponding to data analysis requirements; acquiring target data corresponding to the data field in the HBase storage system according to a user-defined function, wherein the user-defined function is at least used for indicating a query condition for querying the target data corresponding to the data field in the HBase storage system; and carrying out aggregation operation on the target data according to the user-defined function to obtain a data analysis result.
Optionally, the data field includes: the flanksql tense table requires a primary key field and/or a non-primary key field associated with data in the HBase dynamic table.
Optionally, acquiring the target data corresponding to the data field in the HBase storage system includes: and calling the HBase client to execute a query operation according to the target parameters in the user-defined function, wherein the query operation is used for querying the target parameters corresponding to the target parameters in the HBase storage system.
Optionally, the target parameters further include: the HBase dynamic table name of the planned query is stored in the corresponding address of the planned query HBase storage system; invoking the HBase client to perform a query operation includes: establishing connection with a dynamic table object in the HBase storage system according to the address corresponding to the HBase storage system and the name of the HBase dynamic table through the HBase client; and determining the data meeting the query conditions in the user-defined function in the dynamic table object as target data, and reading the target data.
Optionally, the user-defined function is at least used for indicating a mode of aggregating the target data; the aggregation operation of the target data according to the user-defined function comprises the following steps: grouping target data according to the attribute to be aggregated in the aggregation operation to obtain a data set corresponding to the aggregation operation, wherein the data set comprises: data corresponding to the attribute items calculated by the aggregation operation plan in the target data; and executing corresponding aggregation operation on the data set to obtain a data analysis result, wherein the aggregation operation comprises at least one of the following steps: summing, averaging, maximizing, minimizing, and counting.
Optionally, after obtaining the data analysis result, the method further comprises: determining a preset data threshold corresponding to the aggregation operation; and sending target alarm information under the condition that the data analysis result exceeds the range of the corresponding preset data threshold value.
Optionally, determining the user-defined function includes: determining a user-defined function corresponding to the data analysis requirement in a preset function library; and under the condition that the user-defined function corresponding to the data analysis requirement does not exist in the preset function library, responding to the setting instruction of the front-end interactive interface, generating the user-defined function corresponding to the data analysis requirement, and storing the newly generated user-defined function into the preset function library.
According to another aspect of the embodiments of the present application, there is also provided a data processing apparatus, including: the function determining module is used for determining a user-defined function and target parameters corresponding to the user-defined function, wherein the target parameters comprise: data fields corresponding to data analysis requirements; the data query module is used for acquiring target data corresponding to the data field in the HBase storage system according to a user-defined function, wherein the user-defined function is at least used for indicating a query condition for querying the target data corresponding to the data field in the HBase storage system; and the data analysis module is used for carrying out aggregation operation on the target data according to the user-defined function to obtain a data analysis result.
According to still another aspect of the embodiments of the present application, there is also provided an electronic device, including: the system comprises a memory and a processor for running a program stored in the memory, wherein the program executes a data processing method.
According to still another aspect of the embodiments of the present application, there is further provided a nonvolatile storage medium, where the nonvolatile storage medium includes a stored computer program, and a device where the nonvolatile storage medium is located executes the data processing method by running the computer program.
In the embodiment of the application, determining a user-defined function and target parameters corresponding to the user-defined function are adopted, wherein the target parameters comprise: data fields corresponding to data analysis requirements; acquiring target data corresponding to the data field in the HBase storage system according to a user-defined function, wherein the user-defined function is at least used for indicating a query condition for querying the target data corresponding to the data field in the HBase storage system; according to a User-Defined Function, the method for obtaining the data analysis result by carrying out aggregation operation on target data achieves the aim of solving the technical limitation and the like of the non-primary key field associated HBase dynamic table of the FlinkSQL by calling the User-Defined Function (UDF), and further solves the technical problem of poor data processing performance caused by the fact that the related technology needs to issue withdrawal data and state storage when the FlinkSQL temporal table associated HBase dynamic table carries out aggregation analysis.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:
fig. 1 is a block diagram of a hardware architecture of a computer terminal (or electronic device) for implementing a method of data processing according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a method flow of data processing according to an embodiment of the present application;
FIG. 3 is a schematic illustration of a method flow of an aggregation analysis provided in accordance with an embodiment of the present application;
fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.
Detailed Description
In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For the convenience of those skilled in the art to better understand the embodiments of the present application, some technical terms or nouns related to the embodiments of the present application will now be explained as follows:
FlinkSQL: is a high-level data processing technology based on Apache Flink, which provides a way to query and manipulate streaming and batch data using SQL language, allowing users to process data streams using SQL statements, so that real-time data analysis and processing can be performed very conveniently.
HBase: is a columnar storage non-relational database management system running on a Hadoop distributed file system (Hadoop Distributed File System, HDFS) and is suitable for real-time data processing or random read/write access to large amounts of data. HBase is a database suitable for unstructured data storage, unlike a general relational database. Another difference is that HBase is based on column rather than row based patterns.
At present, when streaming data is processed, data of an external HBase storage system is usually associated, the associated data is dynamically changed along with time, and the associated data is processed by service logic and then is sent to a next processing node or a downstream system.
In the related art, when a stream data association HBase dynamic table is processed based on a FlinkSQL, flink-connector-HBase-2.2 version technical stack, right association is not supported in grammar, only static data is supported in conventional association, only HBase main key is supported in association, and therefore the range of processing service requirements is greatly limited.
Specifically, when the related art associates the HBase dynamic TABLE with the FlinkSQL tense TABLE, the principle is that the related art is realized through a user-defined TABLE function UDTF, that is, the flink is connected to an HBase external storage system through an HBase client by calling a UDTF (java program) and returns data to the flink.
However, since the tense table function association typically returns data in one-to-many fashion, business processes typically aggregate operations based on the returned data, requiring more FlinkSQL code to be written, and the aggregate operations of FlinkSQL result in doubling or more of the amount of received data to downstream nodes, as the retired data and state stores are issued, resulting in reduced data processing performance. The scheme has the problems of more manual processing steps, high cost, high fault false alarm rate, poor data processing performance and the like.
In order to solve the above-mentioned problems, related solutions are provided in the embodiments of the present application, and the following detailed description is provided.
In accordance with the embodiments of the present application, there is provided a method embodiment of data processing, it being noted that the steps shown in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order other than that shown.
The method embodiments provided by the embodiments of the present application may be performed in a mobile terminal, a computer terminal, or similar computing device. Fig. 1 shows a block diagram of a hardware structure of a computer terminal (or electronic device) for implementing a data processing method. As shown in fig. 1, the computer terminal 10 (or electronic device) may include one or more processors 102 (shown as 102a, 102b, … …,102 n) which may include, but are not limited to, a microprocessor MCU or a processing device such as a programmable logic device FPGA, a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, the method may further include: a display, an input/output interface (I/O interface), a Universal Serial BUS (USB) port (which may be included as one of the ports of the BUS), a network interface, a power supply, and/or a camera. It will be appreciated by those of ordinary skill in the art that the configuration shown in fig. 1 is merely illustrative and is not intended to limit the configuration of the electronic device described above. For example, the computer terminal 10 may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
It should be noted that the one or more processors 102 and/or other data processing circuits described above may be referred to generally herein as "data processing circuits. The data processing circuit may be embodied in whole or in part in software, hardware, firmware, or any other combination. Furthermore, the data processing circuitry may be a single stand-alone processing module, or incorporated, in whole or in part, into any of the other elements in the computer terminal 10 (or electronic device). As referred to in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of the path of the variable resistor termination to interface).
The memory 104 may be used to store software programs and modules of application software, such as program instructions/data storage devices corresponding to the data processing methods in the embodiments of the present application, and the processor 102 executes the software programs and modules stored in the memory 104, thereby performing various functional applications and data processing, that is, implementing the data processing methods described above. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission means 106 is arranged to receive or transmit data via a network. The specific examples of the network described above may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module for communicating with the internet wirelessly.
The display may be, for example, a touch screen type Liquid Crystal Display (LCD) that may enable a user to interact with a user interface of the computer terminal 10 (or electronic device).
In the above operating environment, the embodiment of the present application provides a data processing method, and fig. 2 is a schematic diagram of a flow of a method for data processing according to the embodiment of the present application, as shown in fig. 2, where the method includes the following steps:
step S202, determining a user-defined function and target parameters corresponding to the user-defined function, wherein the target parameters comprise: data fields corresponding to data analysis requirements;
step S204, obtaining target data corresponding to the data field in the HBase storage system according to a user-defined function, wherein the user-defined function is at least used for indicating a query condition for querying the target data corresponding to the data field in the HBase storage system;
step S206, according to the user-defined function, the target data are aggregated, and a data analysis result is obtained.
According to the method and the device, the problem that performance is reduced due to the fact that the HBase dynamic table technology limit is associated with the non-primary key field of the FlinkSQL tense table and the withdrawn data is issued after aggregation analysis is solved through a user-defined function. The data processing method in step S202 to step S206 in the embodiment of the present application is further described below.
Fig. 3 is a schematic diagram of a method flow of aggregation analysis according to an embodiment of the present application, as shown in fig. 3.
The method comprises the steps of calling a pre-written user-defined function UDF by the FlinkSQL, and determining target parameters input by the user-defined function, wherein the target parameters comprise: and then, inquiring the HBase storage system and returning the associated data by calling the HBase client, performing aggregation analysis by the UDF function, and directly returning the aggregated and analyzed data (namely, the data analysis result) to the FlinkSQL.
In some embodiments of the present application, the data field includes: the flanksql tense table requires a primary key field and/or a non-primary key field associated with data in the HBase dynamic table.
Specifically, the target parameters input by the user-defined function are the stream data fields associated with the data analysis requirements, and support the primary key fields and the non-primary key fields.
As an alternative embodiment, determining the user-defined function comprises the steps of: determining a user-defined function corresponding to the data analysis requirement in a preset function library; and under the condition that the user-defined function corresponding to the data analysis requirement does not exist in the preset function library, responding to the setting instruction of the front-end interactive interface, generating the user-defined function corresponding to the data analysis requirement, and storing the newly generated user-defined function into the preset function library.
Specifically, the user may select a user-defined function with appropriate query conditions and aggregation modes according to actual aggregate analysis requirements, and in this embodiment, the user-defined function may be written by a Java program.
After the user-defined function and the target parameter are determined, the associated data (target data) is obtained from the HBase storage external system, and the specific steps are as follows.
In some embodiments of the present application, obtaining target data corresponding to a data field in an HBase storage system includes the following steps: and calling the HBase client to execute a query operation according to the target parameters in the user-defined function, wherein the query operation is used for querying the target parameters corresponding to the target parameters in the HBase storage system.
As an alternative embodiment, the target parameters further include: the HBase dynamic table name of the planned query is stored in the corresponding address of the planned query HBase storage system; invoking the HBase client to perform the query operation comprises the steps of: establishing connection with a dynamic table object in the HBase storage system according to the address corresponding to the HBase storage system and the name of the HBase dynamic table through the HBase client; and determining the data meeting the query conditions in the user-defined function in the dynamic table object as target data, and reading the target data.
After the target data is acquired, the user-defined function performs aggregation analysis on the target data, and the specific steps are as follows.
In some embodiments of the present application, the user-defined function is at least used to indicate the manner in which the target data is aggregated; according to the user-defined function, the aggregation operation of the target data comprises the following steps: grouping target data according to the attribute to be aggregated in the aggregation operation to obtain a data set corresponding to the aggregation operation, wherein the data set comprises: data corresponding to the attribute items calculated by the aggregation operation plan in the target data; and executing corresponding aggregation operation on the data set to obtain a data analysis result, wherein the aggregation operation comprises at least one of the following steps: summing, averaging, maximizing, minimizing, and counting.
Specifically, according to the defined user-defined function UDF, the aggregate analysis logic of the target data is implemented, for example, by performing operations of calculating, screening, etc. on the data using various aggregate functions (such as summing sum, averaging avg, counting count, etc.). After the aggregation operation is performed, an aggregate analysis result may be obtained, where the result may be a single aggregate value or may be an aggregate result of a group of packets, depending on the definition of the function and the setting of the query, and then the aggregate analyzed data (aggregate analysis result) is directly returned to the FlinkSQL.
As an alternative embodiment, after obtaining the data analysis result, the method further comprises the steps of: determining a preset data threshold corresponding to the aggregation operation; and sending target alarm information under the condition that the data analysis result exceeds the range of the corresponding preset data threshold value.
Specifically, if the aggregate analysis result does not match the expectation, the target alarm information is sent, and the function may need to be debugged and optimized. Problems can be identified by looking at the execution plan, log, and input-output data of the function, and corresponding modifications and improvements can be made.
According to the method and the device, the technical limit of the non-primary key field associated HBase dynamic table of the FlinkSQL is solved through the Java program, the service use scene of a FlinkSQL, HBase technical stack is effectively improved, meanwhile, the writing difficulty of the FlinkSQL of the tense table function scheme is reduced, and the problem of performance degradation caused by data withdrawal after aggregation analysis is solved. The method can effectively improve the service scene used by the FlinkSQL, HBase technical stack, reduce the code cost for developing the FlinkSQL, further improve the maintainability of the program, ensure that the processing performance of the program is not reduced, and meet the real-time service scene of large data volume processing.
Through the steps, the aim of solving the technical limit and other problems of the non-primary key field associated HBase dynamic table of the FlinkSQL tense table is achieved by calling the user-defined function UDF, and the technical problem of poor data processing performance caused by the fact that the related technology needs to issue withdrawal data and state storage when the FlinkSQL tense table associated HBase dynamic table is subjected to aggregation analysis is solved.
According to an embodiment of the present application, there is also provided an embodiment of a data processing apparatus. Fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 4, the apparatus includes:
the function determining module 40 is configured to determine a user-defined function and a target parameter corresponding to the user-defined function, where the target parameter includes: data fields corresponding to data analysis requirements;
in some embodiments of the present application, the data field includes: the flanksql tense table requires a primary key field and/or a non-primary key field associated with data in the HBase dynamic table.
As an alternative embodiment, determining the user-defined function comprises the steps of: determining a user-defined function corresponding to the data analysis requirement in a preset function library; and under the condition that the user-defined function corresponding to the data analysis requirement does not exist in the preset function library, responding to the setting instruction of the front-end interactive interface, generating the user-defined function corresponding to the data analysis requirement, and storing the newly generated user-defined function into the preset function library.
The data query module 42 is configured to obtain target data corresponding to the data field in the HBase storage system according to a user-defined function, where the user-defined function is at least used to indicate a query condition for querying the HBase storage system for the target data corresponding to the data field;
in some embodiments of the present application, obtaining target data corresponding to a data field in an HBase storage system includes the following steps: and calling the HBase client to execute a query operation according to the target parameters in the user-defined function, wherein the query operation is used for querying the target parameters corresponding to the target parameters in the HBase storage system.
As an alternative embodiment, the target parameters further include: the HBase dynamic table name of the planned query is stored in the corresponding address of the planned query HBase storage system; invoking the HBase client to perform the query operation comprises the steps of: establishing connection with a dynamic table object in the HBase storage system according to the address corresponding to the HBase storage system and the name of the HBase dynamic table through the HBase client; and determining the data meeting the query conditions in the user-defined function in the dynamic table object as target data, and reading the target data.
The data analysis module 44 is configured to aggregate the target data according to the user-defined function, so as to obtain a data analysis result.
In some embodiments of the present application, the user-defined function is at least used to indicate the manner in which the target data is aggregated; according to the user-defined function, the aggregation operation of the target data comprises the following steps: grouping target data according to the attribute to be aggregated in the aggregation operation to obtain a data set corresponding to the aggregation operation, wherein the data set comprises: data corresponding to the attribute items calculated by the aggregation operation plan in the target data; and executing corresponding aggregation operation on the data set to obtain a data analysis result, wherein the aggregation operation comprises at least one of the following steps: summing, averaging, maximizing, minimizing, and counting.
As an alternative embodiment, after obtaining the data analysis result, the data analysis module 44 is further configured to: determining a preset data threshold corresponding to the aggregation operation; and sending target alarm information under the condition that the data analysis result exceeds the range of the corresponding preset data threshold value.
The respective modules in the data processing apparatus may be program modules (for example, a set of program instructions for implementing a specific function), or may be hardware modules, and for the latter, they may be represented by the following forms, but are not limited thereto: the expression forms of the modules are all a processor, or the functions of the modules are realized by one processor.
It should be noted that, the data processing apparatus provided in the present embodiment may be used to execute the data processing method shown in fig. 2, so that the explanation of the data processing method is also applicable to the embodiments of the present application, and is not repeated herein.
The embodiment of the application also provides a nonvolatile storage medium, which comprises a stored computer program, wherein the equipment where the nonvolatile storage medium is located executes the following data processing method by running the computer program: determining a user-defined function and target parameters corresponding to the user-defined function, wherein the target parameters comprise: data fields corresponding to data analysis requirements; acquiring target data corresponding to the data field in the HBase storage system according to a user-defined function, wherein the user-defined function is at least used for indicating a query condition for querying the target data corresponding to the data field in the HBase storage system; and carrying out aggregation operation on the target data according to the user-defined function to obtain a data analysis result.
The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments.
In the foregoing embodiments of the present application, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed technology content may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, for example, may be a logic function division, and may be implemented in another manner, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application and are intended to be comprehended within the scope of the present application.
Claims (10)
1. A method of data processing, comprising:
determining a user-defined function and a target parameter corresponding to the user-defined function, wherein the target parameter comprises: data fields corresponding to data analysis requirements;
acquiring target data corresponding to the data field in an HBase storage system according to the user-defined function, wherein the user-defined function is at least used for indicating a query condition for querying the target data corresponding to the data field in the HBase storage system;
and carrying out aggregation operation on the target data according to the user-defined function to obtain a data analysis result.
2. The data processing method according to claim 1, wherein the data field includes: the flanksql tense table requires a primary key field and/or a non-primary key field associated with data in the HBase dynamic table.
3. The method according to claim 2, wherein obtaining target data corresponding to the data field in the HBase storage system includes:
and calling an HBase client to execute a query operation according to the target parameter in the user-defined function, wherein the query operation is used for querying the target parameter corresponding to the target parameter in the HBase storage system.
4. A data processing method according to claim 3, wherein the target parameters further include: the HBase dynamic table name of the planned query is stored in the corresponding address of the planned query HBase storage system; invoking the HBase client to perform a query operation includes:
establishing connection with a dynamic table object in the HBase storage system according to the address corresponding to the HBase storage system and the HBase dynamic table name by the HBase client;
and determining the data meeting the query conditions in the user-defined function in the dynamic table object as the target data, and reading the target data.
5. The data processing method according to claim 1, wherein the user-defined function is at least used to indicate a manner of aggregating the target data; according to the user-defined function, the aggregation operation of the target data comprises the following steps:
grouping the target data according to the attribute to be aggregated of the aggregation operation to obtain a data set corresponding to the aggregation operation, wherein the data set comprises: the data corresponding to the attribute items calculated by the aggregation operation plan in the target data;
and executing the corresponding aggregation operation on the data set to obtain the data analysis result, wherein the aggregation operation comprises at least one of the following steps: summing, averaging, maximizing, minimizing, and counting.
6. The data processing method according to claim 5, wherein after obtaining the data analysis result, the method further comprises:
determining a preset data threshold corresponding to the aggregation operation;
and sending target alarm information under the condition that the data analysis result exceeds the range of the corresponding preset data threshold value.
7. The data processing method of claim 1, wherein determining a user-defined function comprises:
determining the user-defined function corresponding to the data analysis requirement in a preset function library; the method comprises the steps of,
and under the condition that the user-defined function corresponding to the data analysis requirement does not exist in the preset function library, responding to a setting instruction of a front-end interaction interface, generating the user-defined function corresponding to the data analysis requirement, and storing the newly generated user-defined function into the preset function library.
8. A data processing apparatus, comprising:
the function determining module is used for determining a user-defined function and target parameters corresponding to the user-defined function, wherein the target parameters comprise: data fields corresponding to data analysis requirements;
the data query module is used for acquiring target data corresponding to the data field in the HBase storage system according to the user-defined function, wherein the user-defined function is at least used for indicating a query condition for querying the target data corresponding to the data field in the HBase storage system;
and the data analysis module is used for carrying out aggregation operation on the target data according to the user-defined function to obtain a data analysis result.
9. An electronic device, comprising: a memory and a processor for executing a program stored in the memory, wherein the program is executed to perform the data processing method of any one of claims 1 to 7.
10. A non-volatile storage medium, characterized in that the non-volatile storage medium comprises a stored computer program, wherein the device in which the non-volatile storage medium is located performs the data processing method according to any one of claims 1 to 7 by running the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311612925.7A CN117743323A (en) | 2023-11-28 | 2023-11-28 | Data processing method and device, electronic equipment and nonvolatile storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311612925.7A CN117743323A (en) | 2023-11-28 | 2023-11-28 | Data processing method and device, electronic equipment and nonvolatile storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117743323A true CN117743323A (en) | 2024-03-22 |
Family
ID=90251743
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311612925.7A Pending CN117743323A (en) | 2023-11-28 | 2023-11-28 | Data processing method and device, electronic equipment and nonvolatile storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117743323A (en) |
-
2023
- 2023-11-28 CN CN202311612925.7A patent/CN117743323A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11768811B1 (en) | Managing user data in a multitenant deployment | |
| US11797618B2 (en) | Data fabric service system deployment | |
| US11379475B2 (en) | Analyzing tags associated with high-latency and error spans for instrumented software | |
| US10235430B2 (en) | Systems, methods, and apparatuses for detecting activity patterns | |
| CN107229619B (en) | Method and device for counting and displaying calling condition of internet service link | |
| WO2019133763A1 (en) | System and method of application discovery | |
| CN111209309B (en) | Method, device and equipment for determining processing result of data flow graph and storage medium | |
| CN107943841A (en) | Stream data processing method, system and computer-readable recording medium | |
| CN111339073A (en) | Real-time data processing method and device, electronic equipment and readable storage medium | |
| CN112559301B (en) | Service processing method, storage medium, processor and electronic device | |
| CN107133231B (en) | Data acquisition method and device | |
| CN108363741B (en) | Big data unified interface method, device, equipment and storage medium | |
| CN112395333A (en) | Method and device for checking data exception, electronic equipment and storage medium | |
| CN110941634A (en) | Data processing method and device, storage medium and electronic device | |
| CN113535757A (en) | Method, device and equipment for discovering cold and hot data and readable storage medium | |
| CN112861182A (en) | Database query method and system, computer equipment and storage medium | |
| US11574429B1 (en) | Automated generation of display layouts | |
| CN117743323A (en) | Data processing method and device, electronic equipment and nonvolatile storage medium | |
| US20160246783A1 (en) | Systems and methods for managing data related to network elements from multiple sources | |
| CN108241624B (en) | Query script generation method and device | |
| CN111258840A (en) | Cluster node management method and device and cluster | |
| CN110020166A (en) | A kind of data analysing method and relevant device | |
| CN114281549A (en) | Data processing method and device | |
| CN108153834B (en) | Method and device for querying data by commercial intelligent application and electronic equipment | |
| CN112131276A (en) | Data statistics method, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |