CN117406960A - A low-code social data computing platform and device for agile analysis scenarios - Google Patents

Abstract

The invention discloses a method for constructing a low-code data computing platform for a social network agility analysis scene, which comprises 5 main modules: the system comprises a multi-source data access module, a data preprocessing module, a social association diagram construction module, a data analysis module and an analysis result customization output module. The output module can collect corresponding analysis results at regular time to construct specific business output such as portraits and the like. The invention has high decoupling among components, good expansibility, and each module provides various operators for users in advance in a low code form, has flexible change and high development efficiency, and can meet the downstream business of analysis of various social platforms; the method utilizes the emerging technologies such as micro-service functions, stream computing and the like to realize flexible access to the data source and rapid adaptation to the data characteristics, thereby improving the efficiency and quality of data analysis.

Description

A low-code social data computing platform and device for agile analysis scenarios

Technical Field

The present invention relates to the field of computer technology, and in particular to a low-code social data computing platform and device for agile analysis scenarios.

Background Art

With the rapid development of social platforms, the massive amount of data generated by a large number of users has brought challenges to data analysis. In addition, the heterogeneity of data sources and the constant changes in data characteristics have also brought certain difficulties to data analysis. For specific application scenarios, it is often necessary to rewrite business code. However, the data processing of these business scenarios often has similarities.

Most of the existing social platform data analysis systems and methods use traditional data analysis architectures and technologies, which often require a lot of programming and debugging work and require a high level of developer skills. Therefore, a low-code social platform data analysis processing method and system is needed to improve the efficiency and quality of data analysis while reducing development costs and time.

Summary of the invention

To this end, the present invention first proposes a low-code social data computing platform and device for agile analysis scenarios, which are applied to social data analysis systems, and are characterized in that the system includes a data source definition process and a data source processing process;

The data source definition process defines the data source data of the social platform, which includes 4 modules:

The data format definition module defines the data format of heterogeneous social platforms and performs field binding for the relationships and attributes unique to social platforms.

Preprocessing flow definition module, the user defines the flow processing flow of the data source data preprocessing module by arranging several predefined data preprocessing functions or adding custom preprocessing functions;

Analysis task arrangement module: users select different analysis services based on the analysis services provided by the system analysis service platform according to different analysis results, and combine and arrange the results;

Output result definition module, which defines the format and content of output results according to user requirements;

The data source processing flow runs the defined processing flow and includes 5 modules:

The data access module uses the registration callback mechanism to achieve flexible access to the data source in the form of microservice functions, accesses the data defined by the data format definition module to the system, performs preliminary cleaning and formatting on the received data, and sends the data to the data preprocessing module for further processing;

The data preprocessing module preprocesses the continuously increasing and disordered social platform data in a streaming computing organization mode based on the functions defined in the preprocessing stream definition module;

The social association graph construction module uses the message subscription framework to input the data preprocessing module to preprocess the data, decouple the data storage process from the processing process, construct the data into a social association graph, and store it in the graph database;

The data analysis module inputs the social association graph based on the selection of the analysis task orchestration module and customizes the output of the data analysis results;

The customized output module is based on the definition of the output result definition module. According to the user's analysis needs, it registers the corresponding data analysis module, regularly collects the corresponding analysis results, and constructs specific business outputs.

The data source is edited and configured in a low-code manner using the drag-and-drop editing page provided by the control panel. For each newly created data source, the control panel will create a corresponding data source controller to uniformly manage the creation, query, update, and deletion of the data source. The data source controller will periodically send heartbeat signals to the control panel to report its own status, allowing the data source manager to take timely response measures to the loss of connection with the controller.

After being created, the data source controller notifies the resource coordinator to allocate resources to specific process modules. After notifying the resource coordinator to create a function, the data source A controller creates a preprocessing function 1 and a processing function m on physical node 1 and physical node n, respectively. There can be multiple processing functions on a physical node. The created function will periodically send heartbeat signals to the subordinate manager to inform the function and the function's running container of the running status, and perform capacity expansion and contraction according to the load conditions.

The technical effects to be achieved by the present invention are:

The low-code data computing platform construction method for social network agile analysis scenarios provided by the present invention can agilely adapt to rapidly changing analysis business needs for social networks, and based on massive, heterogeneous social platform data, it carries out business involvement in the form of low code, and performs efficient and rapid processing and analysis, and finally outputs customized analysis results to meet the downstream business of various social platform analyses. The method includes a multi-source data access module, a data preprocessing module, a social association graph construction module, a data analysis module, and an analysis result customized output module. Through the high degree of decoupling of these modules and the provision of operators in the form of low code, users can process and analyze data quickly and flexibly. At the same time, the method utilizes emerging technologies such as microservice functions and streaming computing to achieve flexible access to data sources and rapid adaptation to data features, thereby improving the efficiency and quality of data analysis.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1: Architecture of the low-code data computing platform construction method for agile analysis scenarios;

Figure 2 Data source creation process;

Figure 3 Data source controller architecture;

Figure 4 shows the overall data analysis and calculation process;

Figure 5: The overall analysis and calculation process of the data.

DETAILED DESCRIPTION

The following is a preferred embodiment of the present invention and in conjunction with the accompanying drawings, further describing the technical solution of the present invention, but the present invention is not limited to this embodiment.

The present invention proposes a method for constructing a low-code data computing platform for agile analysis scenarios.

Specifically, the present invention discloses a low-code data computing platform construction system for social network agile analysis scenarios, which includes a data source definition process and a data source processing process for each data source:

Specifically, the data source definition process defines the data source data of the social platform, which includes 4 modules:

The data format definition module defines the data formats of heterogeneous social platforms, including but not limited to text, image, video and other data formats. In addition to field binding for several relationships (likes, reposts and comments) and several attributes (account ID, user nickname) that are unique to social platforms, the data types and meanings of other fields are defined by users.

The example format is as follows:

The system-defined fields are the minimum set of basic attributes of social platform data, including the minimum set of information required to build a correlation graph:

text_fields: defines the text fields. The example contains only one content field, whose type is "text".

image_fields: defines the image fields. The example only contains one image_url field, whose type is "image".

video_fields: defines the video fields. The example only contains one video_url field, whose type is "video".

relation_fields: defines relation fields, including like, comment, and share. In the example, the like, comment, and share fields are defined respectively, and their types are all "relation". These relation fields can be bound to other related fields (such as account_id and user_name).

User-defined fields can be used as additional information for data analysis to enhance the authenticity and reliability of analysis results:

user_defined_fields: defines user-defined fields. In the example, two custom fields, custom_field1 and custom_field2, are defined, and their types are "text" and "number" respectively. Users can define more custom fields as needed.

After adopting the above data definition, the platform can uniformly access data from multiple sources, fully decouple the data source from subsequent business, and enable complex analysis business.

In the preprocessing flow definition module, users define the flow processing flow of the data source data preprocessing module by arranging several predefined data preprocessing functions or adding custom preprocessing functions. The specific methods include but are not limited to data cleaning, deduplication, filtering, normalization and other processing processes to ensure the accuracy and efficiency of subsequent data processing and analysis. The configuration example is as follows:

Pipeline is an array of preprocessing streams, defining a series of preprocessing functions and their parameters.

Each preprocessing function is represented by an object, which contains two fields: name and params. The name field indicates the name of the preprocessing function, such as "dataCleaning", "dataDecoding", and "dataFiltering". The params field is an object used to set the parameters of the preprocessing function. The specific parameters depend on each function.

In the example, the "dataCleaning" function has two parameters: removeStopwords and removePunctuation, which determine whether to remove stop words and punctuation.

The "dataDeduplication" function has one parameter: keyField, which specifies the key field used for deduplication.

The "dataFiltering" function has one parameter: filterCondition, which sets the filtering condition, for example, filtering out data with more than 1000 likes.

In the analysis task orchestration module, users can select different analysis services based on different analysis needs and combine and orchestrate the results based on the analysis services provided by the system analysis service platform. This module supports the orchestration of multiple analysis tasks and runs them in a serverless manner. These analysis service algorithms can be classified according to different categories, including but not limited to machine learning, deep learning, and user-defined algorithms.

For machine learning algorithms, support is provided for Support Vector Machine, Random Forest, Logistic Regression, K-Nearest Neighbors, Principal Component Analysis, AdaBoost, XGBoost, etc.

For deep learning algorithms, support is provided for Convolutional Neural Network, Recurrent Neural Network, Long Short-Term Memory, Bidirectional Recurrent Neural Network, Generative Adversarial Network, Transformers, and Deep Reinforcement Learning Algorithms.

For user-defined algorithms, users can develop their own algorithms for analysis based on their own needs and data characteristics. These algorithms can be based on traditional statistical methods, knowledge models in specific fields, or other customized analysis methods.

The analysis task orchestration module supports users to select different algorithms and configure their parameters to meet specific analysis needs. Users can flexibly select suitable algorithms based on data characteristics, problem types and business scenarios, and combine and orchestrate multiple algorithm results to obtain the required analysis results. This can achieve multi-angle and multi-dimensional analysis of data, providing users with more comprehensive and accurate analysis services. The configuration example is as follows:

analysisTasks is an array of analysis tasks, defining a series of analysis tasks and their algorithms and parameters. Each analysis task is represented by an object, which contains three fields: name, algorithm and params.

The name field is the name of the analysis task, such as "sentimentAnalysis" and "topicClassification".

The algorithm field specifies the algorithm category to be used, such as "machineLearning", "deepLearning" or other user-defined algorithms.

The params field is an object used to set the parameters of the analysis task. The specific parameters depend on each algorithm.

In the example, the "sentimentAnalysis" analysis task uses a machine learning algorithm to perform sentiment analysis. It analyzes the "text" field and stores the results in the "sentiment" field.

The "topicClassification" analysis task uses a deep learning algorithm for topic classification. It analyzes the "text" field and stores the results in the "topic" field.

The "entityExtraction" analysis task uses a natural language processing algorithm for entity extraction. It analyzes the "text" field and stores the results in the "entities" field.

The "networkAnalysis" analysis task uses a graph processing algorithm to perform social network analysis. It analyzes fields such as "accountID", "likes", "comments", and "shares" and stores the results in the "networkMetrics" field.

Based on actual needs, you can customize the analysis tasks, algorithms, and parameters according to this example, and set the fields to be analyzed as needed.

Output result definition module: Define the format and content of output results according to user needs, including but not limited to statistical reports, charts, social association diagrams, etc. This module supports multiple output result definitions, multiple data display methods, and multiple data exposure methods to meet different business needs. The configuration example is as follows:

outputResults is an array of output results, defining a series of output results and their types and fields. Each output result is represented by an object, which contains type and corresponding fields.

The type field specifies the type of output result, such as "statistics", "chart", or "socialGraph".

In the example, the first output is of type "statistics", which generates statistics, calculated over the specified fields, in this case "sentiment" and "topic".

The second output result is of type "chart", which generates a bar chart using the "topic" field as the X-axis and the "sentiment" field as the Y-axis.

The third output result type is "socialGraph", which generates a social relationship graph and uses the "entities" field as node information and the "networkMetrics" field as link information.

Based on actual needs, you can customize the output results and their types and related fields according to this example.

Specifically, the data source processing flow runs the defined processing flow and includes 5 modules:

Data access module: This module uses the registration callback mechanism to achieve flexible access to data sources in the form of microservice functions, and connects social platform data from different sources to the system. The access module will perform preliminary cleaning and formatting on the received data, and send the data to the data preprocessing module for further processing;

Data preprocessing module: This module uses streaming computing as the main organizational model to preprocess the continuously increasing and disordered social platform data. It can quickly adapt to the newly added data features by adding streaming processing links. Its pseudo code is as follows:

Social association graph construction module: This module decouples the data storage process from the processing process through the message subscription framework, constructs the data into a social association graph, and stores it in the graph database. This helps to convert social platform data into graphical data, facilitating subsequent data analysis and customized output;

Data analysis module: This module achieves high scalability of analysis capabilities in the form of a serverless service. It can flexibly select various data analysis algorithms and provide preset low-code operators and visualization components, so that users can easily customize the output of data analysis results;

The above pseudo code defines the AnalysisModule interface. All data analysis modules need to implement this interface and provide the Analyze method to execute specific data analysis logic. Then two data analysis services are called, SentimentAnalysis and TopicClassification, which perform sentiment analysis and topic classification on the data respectively. Finally, in the CombineAnalysisModules function, different analysis modules are combined and their Analyze methods are called in sequence to complete the data analysis. Different analysis modules can be created according to actual needs and passed into the CombineAnalysisModules function together with the original data for analysis. Finally, the system can process and output the analysis results.

Customized output module: This module registers the corresponding data analysis module based on the user's analysis needs, and regularly collects the corresponding analysis results to build specific business outputs such as portraits to enable downstream businesses. The output module also supports customized output formats and channels to meet the output needs of different users.

Among them, the data source definition process and the data source processing process are interrelated and correspond to each other, the data format definition module configures the multi-source data access module, the processing flow definition module configures the data preprocessing module, the analysis task orchestration module configures the data analysis module, and the output result definition module configures the customized output module and configures the acceptance scheme of the downstream tasks.

Among them, the data source is edited and configured in a low-code manner through the drag-and-drop editing page provided by the control panel, and users do not need to perform detailed coding operations for data flow control.

For each newly created data source, the control panel will create a corresponding data source controller to centrally manage the creation, query, update, and deletion of the data source. The data source controller will periodically send heartbeat signals to the control panel to report its own status, so that the data source manager can take timely response measures if the controller loses connection.

After the data source controller is created, it will notify the resource coordinator to allocate resources to specific process modules. Taking the preprocessing manager as an example, the resource coordinator searches for suitable idle physical nodes in the cluster based on the default number of functions set in the preprocessing manager and creates preprocessing functions on them.

As shown in Figure 2 below, after notifying the resource coordinator to create a function, the resource data source A controller creates a pre-processing function 1 and a processing function m on physical node 1 and physical node n, respectively. At the same time, there can be multiple processing functions on a physical node.

The expansion function is designed to dynamically adjust resources according to load conditions, so that the system can adapt to different workloads and improve resource utilization efficiency.

When resource usage exceeds the threshold, resource_usage>threshold, you can set the threshold based on actual conditions or determine it through tuning; if it is lower than the threshold, there is no need to expand capacity.

Calculate the number of instances that need to be expanded: num_instances = int (resource_usage/threshold) + 1. Calculate the number of instances that need to be expanded based on the ratio of resource usage to the threshold.

The number of calculated instances is limited to not exceed the maximum number of instances: num_instances = min(num_instances, max_instances).

Returns the calculated number of instances, indicating the number of instances that need to be expanded. The code example is as follows:

The expansion function can be adjusted and modified according to the actual situation. The resource utilization threshold, maximum number of instances and other parameters can be set according to system requirements and actual conditions to meet the needs and improve system performance.

The created function will periodically send heartbeat signals to the subordinate manager to inform the function and the running status of the function's container, and perform capacity expansion and contraction based on the load conditions.

In particular, the data analysis platform can be understood as a set of standardized algorithm libraries organized in a serverless form. Since its computing power may rely on dedicated devices such as graphics computing components, it can be deployed separately in another set of hardware equipment environments. The data processing flow and the service platform are coordinated through communication protocols, so that the components can be fully decoupled and the overall robustness and availability can be improved;

Figure 3 introduces the overall data analysis and calculation process after the data source is defined.

Step 1: Data source A pushes the acquired message to the message queue using the API provided by the software or the general message queue SDK. Messages can be pushed in the form of single data text or in batches.

Step 2: The function instance under the data source A manager will regularly pull new data from the message queue, and after modifying the topic of the message, leave it for subsequent functions to process. Finally, the entire preprocessing process is executed in sequence.

Step 3: The processed data will wait in the message queue for the function instance of the social association graph builder to capture. The attribute vector embedding function will normalize the attribute fields in the data according to the pre-defined types and convert them into vector form. After entering the database, it will be used by the subsequent analysis module. The data will then be organized in the form of a graph and stored in the graph database. Together, they provide underlying data support for subsequent analysis services.

Step 4: The data analysis manager will periodically call the analysis algorithm provided by the corresponding data analysis platform based on the configured indicators. Analysis tasks are generally divided into two types:

If the analysis indicator is of an independent type, that is, it does not need to rely on other data, but only relies on statistical values such as max, it directly updates the indicator of the corresponding node in the graph database after calculation.

If the analysis indicator depends on other data, such as the need to roam the entire graph, and the addition of new nodes will bring updates to other node indicators. Such computing tasks usually take a long time to calculate, and the computing tasks will be triggered when the overall cluster load is low, the timing or accumulation of the same tasks reaches the counter threshold. When updating the database, the database will be locked and the service will be provided again after the update is completed.

Step 5: The output manager will periodically reorganize the analysis indicator data in the form of configuration requirements and finally provide it to downstream applications. There are two general methods:

If the user configures the passive receiving mode, the output manager will regularly organize the results and send them to the message queue, or call the hook function configured by the user to send the results to the corresponding address.

If the user configures the active receiving mode, the user only needs to set the call API provided by the system to obtain the latest analysis results.

Repeat steps 1 to 5.

Claims (3)

1. A low-code social data computing platform and a device of an agile analysis scene are applied to a social data analysis system, and are characterized in that the system comprises a data source definition flow and a data source processing flow;

the data source definition flow defines social platform data source data and comprises 4 modules:

the data format definition module is used for defining the data format of the heterogeneous social platform and carrying out field binding on the specific relationship and attribute of the social platform;

the preprocessing flow definition module is used for defining the flow processing flow of the data source data preprocessing module by arranging a plurality of predefined data preprocessing functions or adding a user-defined preprocessing function by a user;

the analysis task scheduling module is used for selecting different analysis services according to different analysis results based on analysis services provided by the system analysis service center, and combining and scheduling the results;

the output result definition module is used for defining the format and the content of the output result according to the requirements of the user;

the data source processing flow runs the defined processing flow and comprises 5 modules:

the data access module utilizes a registration callback mechanism to realize flexible access to a data source in the form of a micro service function, performs preliminary cleaning and formatting on received data by the data access system defined by the data format definition module, and sends the data to the data preprocessing module for further processing;

the data preprocessing module is used for preprocessing continuously-added and unordered social platform data in a streaming calculation organization mode based on the function defined by the preprocessing flow definition module;

the social connection diagram construction module inputs data to be preprocessed by the data preprocessing module through the message subscription frame, decouples the data storage flow from the processing flow, constructs the data into a social connection diagram and stores the social connection diagram into the diagram database;

the data analysis module inputs the social association graph based on the selection of the analysis task arrangement module, and customizes and outputs a data analysis result;

the customized output module registers a corresponding data analysis module according to the analysis requirement of a user based on the definition of the output result definition module, and periodically collects corresponding analysis results to construct specific business output.

2. The low-code social data computing platform and apparatus of a agile analysis scenario of claim 1, wherein: the drag editing page provided by the control panel of the data source is edited and configured in a low-code mode, the control panel can create a corresponding data source controller for each newly-built data source, the creation, inquiry, update and deletion of the data source are uniformly managed, and the data source controller can periodically send heartbeat signals to the control panel to report the state of the data source controller, so that the data source manager can take timely response measures for the condition of the controller disconnection.

3. The low-code social data computing platform and apparatus of a agile analysis scenario of claim 2, wherein: after the data source controller is established, the resource coordinator is informed to allocate resources to a specific flow module, after the data source controller is informed to the resource coordinator to establish functions, the data source A controller establishes a preprocessing function 1 and a processing function m on a physical node 1 and a physical node n respectively, a plurality of processing functions can exist on one physical node, the established functions can send heartbeat signals to a subordinate manager at regular time, the functions and the running states of a function running container are informed, and capacity expansion and contraction processing is carried out according to load conditions.