CN118051566A - Supply chain financial user tag management system - Google Patents

Abstract

The application provides a supply chain financial user tag management system, wherein a user tag management platform in the system is used for acquiring tag definition data; the background application subsystem is used for sending the tag definition data to a relational database for storage and sending a tag task to the Spark computing module; the Spark computing module is used for retrieving the user tag types and the user tag setting rules of the tag task, acquiring the raw data of the financial users of the supply chain from the offline data warehouse to determine the user tags, and storing the user tags into the Doris tag library. The application is based on Spark and Doris architecture, can effectively improve the development efficiency and application reliability of the system, improve the resource utilization rate, ensure the data consistency and maintainability of the management system, effectively improve the efficiency and effectiveness of acquiring and storing the labels of the financial users of the supply chain, and further improve the real-time property of label inquiry, the running stability and the use flexibility of the management system.

Description

Supply chain financial user tag management system

Technical Field

The application relates to the technical field of big data processing, in particular to a supply chain financial user tag management system.

Background

At present, supply chain finance has become one of the points of force and break for transformation by financial related institutions such as commercial banks. Supply chain financial user management has become a viable support and profit source for enterprises, and almost all enterprise administrators recognize that supply chain financial management plays a significant role in enterprise strategy. The supply chain financial users to and from the enterprise network are often a huge data group, so the user tags need to be set for the supply chain financial users. The user tag is an attribute description of a certain dimension characteristic of a user, can accurately describe the characteristic of a certain user or a certain class of user, is computer-readable structured data, and can be directly applied to actual service scenes.

However, the conventional user tag management method for the supply chain financial users is usually implemented by using scripts, and although the method is simple and easy to use, the method has the problems of low development efficiency, insufficient resource utilization, low efficiency in processing large-scale data and complex computing tasks, incapability of ensuring data consistency, difficult code maintenance and the like.

Disclosure of Invention

In view of this, embodiments of the present application provide a supply chain financial user tag management system that obviates or mitigates one or more of the disadvantages of the prior art.

One aspect of the present application provides a supply chain financial user tag management system comprising: the system comprises a background application subsystem, a user tag management platform and a big data cluster subsystem which are respectively in communication connection with the background application subsystem;

The big data cluster subsystem is also provided with a relational database, a Spark computing module and a Doris tag library;

The user tag management platform is used for acquiring tag definition data, and the tag definition data comprises: corresponding relations among various user label types, user label setting rules and label tasks of the supply chain financial users;

The background application subsystem is used for sending the label definition data received from the user label management platform to the relational database as metadata for storage, and sending the label task to the Spark computing module;

The Spark computing module is used for retrieving each user tag type and user tag setting rule corresponding to the tag task from the relational database when or after the tag task is received, acquiring the raw data of the supply chain financial user corresponding to the tag task from an offline data warehouse, further determining each user tag of each supply chain financial user corresponding to the raw data of the supply chain financial user according to each user tag type and user tag setting rule corresponding to the tag task, and sending each user tag of each supply chain financial user to the Doris tag library;

the Doris tag library is used for storing received user tags of the supply chain financial users.

In some embodiments of the application, the user tag management platform comprises: a user tag management module and an interface service management module;

the user tag management module is used for receiving the tag definition data and sending the tag definition data to the background application subsystem;

The interface service management module is used for receiving a target user tag query request and sending the target user tag query request to the background application subsystem;

Correspondingly, the background application subsystem is further used for inquiring user tag inquiry result data of the target supply chain financial user corresponding to the target user tag inquiry request from the Doris tag library based on a preset unified service interface platform when or after the target user tag inquiry request is received, and sending the user tag inquiry result data to the interface service management module for the corresponding inquiry user to check.

In some embodiments of the present application, the user tag management platform further comprises: a label task scheduling management module;

The tag task scheduling management module is used for retrieving the tag definition data from the background application subsystem, setting Spark flow tasks taking Quartz as task scheduling frames according to the tag definition data, and then sending scheduling and monitoring instructions aiming at the tag tasks to the background application subsystem by the Spark flow tasks so that the background application subsystem sends the tag tasks to the Spark calculation module according to execution time, frequency and priority appointed by the scheduling and monitoring instructions of the tag tasks, and the Spark calculation module is used for generating user tags of each supply chain financial user corresponding to the tag tasks.

In some embodiments of the present application, the user tag management platform further comprises: a user grouping management module;

the user grouping management module is used for receiving the user grouping rule and sending the user grouping rule to the background application subsystem;

Correspondingly, the background application subsystem is further configured to retrieve respective user tags of the supply chain financial users from the Doris tag library, perform grouping processing on the supply chain financial users according to the respective user tags of the supply chain financial users and the user grouping rules, so as to obtain corresponding crowd packages, and then send the crowd packages to the Doris tag library for storage.

In some embodiments of the present application, the interface service management module is further configured to receive a target crowd-sourced query request, and send the target crowd-sourced query request to the background application subsystem;

Correspondingly, the background application subsystem is further used for inquiring crowd-sourced inquiry result data corresponding to the target crowd-sourced inquiry request from the Doris tag library based on a preset unified service interface platform when or after the target crowd-sourced inquiry request is received, and sending the crowd-sourced inquiry result data to the interface service management module for the corresponding inquiry user to check.

In some embodiments of the present application, the user tag management platform further comprises: a task monitoring management module;

The task monitoring management module is used for calling the execution state data of the current tag task execution of the Spark computing module through the background application subsystem based on a preset plug-in, and displaying the execution state data.

In some embodiments of the present application, the user tag management platform further comprises: an abnormality early warning management module;

The abnormal early warning management module is in communication connection with the task monitoring management module, so that corresponding abnormal early warning prompt information is sent out when the execution state data acquired by the task monitoring management module is abnormal.

In some embodiments of the present application, the user tag management platform further comprises: a user rights management module;

The user authority management module is used for receiving user information of a management user and/or a query user, and determining user authority corresponding to the user information based on prestored user authority comparison data so as to carry out authority management on the management user and/or the query user.

In some embodiments of the application, the background application subsystem is further communicatively coupled to an external application external to the supply chain financial user tag management system;

correspondingly, the background application subsystem is further used for extracting the user labels of the supply chain financial users from the Doris label library, generating visual data corresponding to the user labels of the supply chain financial users, and then sending the visual data to the external application and/or the user label management platform.

In some embodiments of the present application, the background application subsystem is previously built based on a Spring Boot frame, and the user tag management platform is previously built based on a vue.

The application provides a supply chain financial user tag management system which is provided with a background application subsystem, a user tag management platform and a big data cluster subsystem, wherein the user tag management platform and the big data cluster subsystem are respectively in communication connection with the background application subsystem; the big data cluster subsystem is also provided with a relational database, a Spark computing module and a Doris tag library; the user tag management platform is used for acquiring tag definition data, and the tag definition data comprises: corresponding relations among various user label types, user label setting rules and label tasks of the supply chain financial users; the background application subsystem is used for sending the label definition data received from the user label management platform to the relational database as metadata for storage, and sending the label task to the Spark computing module; the Spark computing module is used for retrieving each user tag type and user tag setting rule corresponding to the tag task from the relational database when or after the tag task is received, acquiring the raw data of the supply chain financial user corresponding to the tag task from an offline data warehouse, further determining each user tag of each supply chain financial user corresponding to the raw data of the supply chain financial user according to each user tag type and user tag setting rule corresponding to the tag task, and sending each user tag of each supply chain financial user to the Doris tag library; the Doris tag library is used for storing received user tags of the supply chain financial users, and based on Spark and Doris architecture, the application can effectively improve the development efficiency and application reliability of the system, improve the resource utilization rate, ensure the data consistency and maintainability of the management system, effectively improve the acquisition and storage efficiency and effectiveness of the supply chain financial user tags, and further improve the real-time performance of tag inquiry, the running stability and the use flexibility of the management system.

Additional advantages, objects, and features of the application will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and drawings.

It will be appreciated by those skilled in the art that the objects and advantages that can be achieved with the present application are not limited to the above-described specific ones, and that the above and other objects that can be achieved with the present application will be more clearly understood from the following detailed description.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate and together with the description serve to explain the application. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the application. Corresponding parts in the drawings may be exaggerated, i.e. made larger relative to other parts in an exemplary device actually manufactured according to the present application, for convenience in showing and describing some parts of the present application. In the drawings:

FIG. 1 is a schematic diagram of a first architecture of a supply chain financial user tag management system in accordance with one embodiment of the present application.

FIG. 2 is a schematic diagram of a second architecture of a supply chain financial user tag management system in accordance with one embodiment of the present application.

FIG. 3 is a schematic diagram of a third architecture of a supply chain financial user tag management system in accordance with an embodiment of the present application.

FIG. 4 is a functional logic diagram of a supply chain financial user tag management system in accordance with one embodiment of the present application.

FIG. 5 is a schematic diagram of a user tag platform architecture corresponding to a supply chain financial user tag management system according to an embodiment of the present application.

FIG. 6 is a flow chart illustrating an implementation of the user tag management platform in the supply chain financial user tag management system according to an embodiment of the present application.

Detailed Description

The present application will be described in further detail with reference to the following embodiments and the accompanying drawings, in order to make the objects, technical solutions and advantages of the present application more apparent. The exemplary embodiments of the present application and the descriptions thereof are used herein to explain the present application, but are not intended to limit the application.

It should be noted here that, in order to avoid obscuring the present application due to unnecessary details, only structures and/or processing steps closely related to the solution according to the present application are shown in the drawings, while other details not greatly related to the present application are omitted.

It should be emphasized that the term "comprises/comprising" when used herein is taken to specify the presence of stated features, elements, steps or components, but does not preclude the presence or addition of one or more other features, elements, steps or components.

It is also noted herein that the term "coupled" may refer to not only a direct connection, but also an indirect connection in which an intermediate is present, unless otherwise specified.

Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

Conventional tag implementations are typically implemented using scripts, and although this approach is simple and easy to use, it suffers from the following drawbacks:

Developer inefficiency: traditional script implementations may require writing a large amount of code, including data processing logic, error handling, resource management, etc., which can cause developers to spend more time and effort on coding, reducing development efficiency.

The communication cost is high: because script implementation usually requires manual writing of a large amount of codes, and the readability is poor, the difficulty of understanding and collaboration among team members is increased, and the communication cost is high.

Lack of modularity and multiplexing: script implementations typically employ top-down programming, lacking good modularity and reusability. If multiple tasks require similar computational logic, often repeated writing of code is required, increasing maintenance costs.

Resource utilization is insufficient: script implementations typically run on a single machine, failing to fully utilize distributed computing resources. This may result in inefficiency in computing tasks that do not fully exploit the benefits of clustered computing.

Data consistency cannot be guaranteed: under traditional script implementations, there may be a problem in that data consistency cannot be guaranteed. Particularly in a multi-step computing process, if one step is faulty or needs to be rolled back, data inconsistency of the entire computing process may be caused.

Poor fault tolerance: script implementations typically have poor support for error handling and fault tolerance, and once errors occur during computation, manual intervention may be required to repair and recover, increasing the difficulty of system maintenance.

Parallel computing is difficult: under conventional script implementations, threads or processes often need to be manually managed to implement parallel computing, which increases complexity and maintainability of code, and also tends to introduce concurrency security issues.

Resource management is difficult: script implementations typically require manual management of allocation and release of computing resources, which may be problematic with uneven resource utilization or wasteful resources.

Monitoring and scheduling are inconvenient: in conventional script implementations, monitoring the execution state of computing tasks, scheduling the execution order of tasks, etc. may require additional effort and is often not flexible and automated.

In order to solve the above problems, the embodiment of the application provides a supply chain financial user label management system, which is based on a Spark and Doris architecture, and aims to solve the problems of low development efficiency, low efficiency, difficult code maintenance and the like of the traditional script implementation mode when processing large-scale data and complex computing tasks.

The following examples are provided to illustrate the invention in more detail.

Based on this, an embodiment of the present application provides a supply chain financial user tag management system, referring to fig. 1, which specifically includes the following contents:

The system comprises a background application subsystem 20, a user tag management platform 10 and a big data cluster subsystem 30 which are respectively in communication connection with the background application subsystem 20.

The big data cluster subsystem 30 is further provided with a relational database 31, a Spark computing module 32 and a Doris tag library 33.

It can be appreciated that Spark is a powerful open source distributed computing framework with high speed, versatility and ease of use. In a user tag management system, spark is used for large-scale data processing and distributed computing to achieve fast and efficient data processing capabilities. Through the distributed computing power of Spark, a large amount of user data can be processed and analyzed in parallel, and the processing speed and throughput of the system are improved. User data in the supply chain financial industry can also be mined and analyzed using Spark's machine learning and data mining functions to generate data mining class labels.

The user tag management platform 10 is configured to obtain tag definition data, where the tag definition data includes: corresponding relations among various user label types, user label setting rules and label tasks of the supply chain financial users;

The background application subsystem 20 is configured to send the tag definition data received from the user tag management platform 10 as metadata to the relational database 31 for storage, and send the tag task to the Spark computing module 32.

The Spark calculation module 32 is configured to retrieve, from the relational database 31, the user tag type and the user tag setting rule corresponding to the tag task when or after the tag task is received, and obtain, from an offline data warehouse, supply chain financial user raw data corresponding to the tag task, and further determine, according to the user tag type and the user tag setting rule corresponding to the tag task, a user tag of each supply chain financial user corresponding to the supply chain financial user raw data, and send the user tag of each supply chain financial user to the Doris tag library 33.

The Doris tag library 33 is used to store the received respective user tags for the respective supply chain financial users.

It will be appreciated that Doris (under the name Palo) is a high-performance, scalable column store analytical database. In the user tag management system, the user tag data is stored and managed using Doris as an underlying database. Doris has the capability of high-speed query and high concurrency processing, and can meet the storage and access requirements of large-scale data in the supply chain financial industry. Through the column storage characteristic of Doris, efficient data compression and query optimization can be realized, and the performance and response speed of the system are improved.

Under Spark and Doris architecture, spark can be used to clean, preprocess and extract features of the original data, and then the calculated tag data is imported into Doris for real-time OLAP query and analysis. The advantage of this architecture is that the combination of the powerful data processing capability of Spark and the real-time query analysis capability of Doris enables efficient data processing and real-time data query analysis.

As can be seen from the above description, the supply chain financial user tag management system provided by the embodiment of the present application is based on Spark and Doris architecture, which can effectively improve the development efficiency and application reliability of the system, improve the resource utilization rate, ensure the data consistency and maintainability of the management system, and effectively improve the efficiency and effectiveness of acquiring and storing the supply chain financial user tag, so as to improve the real-time performance of tag query, and the operation stability and the use flexibility of the management system.

In order to further improve the real-time performance and effectiveness of the user tag query in the supply chain financial user tag management, in the supply chain financial user tag management system provided by the embodiment of the present application, referring to fig. 2, the user tag management platform 10 in the supply chain financial user tag management system specifically includes the following contents:

A user tag management module 11 and an interface service management module 12.

The user tag management module 11 is configured to receive the tag definition data and send the tag definition data to the background application subsystem 20.

It will be appreciated that the user tag management module 11 forms tag definition data by configuring tag rules for specifying conditions, algorithms, output formats, etc. for tag calculation. The tag definition data includes a tag name, a tag type, a tag calculation mode, and the like.

The interface service management module 12 is configured to receive a target user tag query request and send the target user tag query request to the background application subsystem 20.

Correspondingly, the background application subsystem 20 is further configured to query, based on a preset unified service interface platform, the user tag query result data of the target supply chain financial user corresponding to the target user tag query request from the Doris tag library 33, and send the user tag query result data to the interface service management module 12 for viewing by a corresponding querying user when or after receiving the target user tag query request.

In order to further improve the reliability and effectiveness of label task execution in the supply chain financial user label management, in the supply chain financial user label management system provided by the embodiment of the present application, referring to fig. 2, the user label management platform 10 in the supply chain financial user label management system further specifically includes the following contents:

the label task schedule management module 13.

The tag task scheduling management module 13 is configured to retrieve the tag definition data from the background application subsystem 20, set a Spark flow task with a quantiz as a task scheduling framework according to the tag definition data, and then send a scheduling and monitoring instruction for the tag task to the background application subsystem 20 by using the Spark flow task, so that the background application subsystem 20 sends the tag task to the Spark calculation module 32 according to the execution time, frequency and priority specified by the scheduling and monitoring instruction of the tag task, so that the Spark calculation module 32 generates the user tag of each supply chain financial user corresponding to the tag task.

Among these, quartz is an open source job scheduling framework, which is written entirely in Java and designed for J2SE and J2EE applications.

In order to further improve the reliability and effectiveness of user grouping in the supply chain financial user tag management, in the supply chain financial user tag management system provided by the embodiment of the present application, referring to fig. 2, the user tag management platform 10 in the supply chain financial user tag management system further specifically includes the following contents:

A user grouping management module 14.

The user grouping management module 14 is configured to receive a user grouping rule and send the user grouping rule to the background application subsystem 20;

Correspondingly, the background application subsystem 20 is further configured to retrieve the user tag of each supply chain financial user from the Doris tag library 33, perform grouping processing on each supply chain financial user according to the user tag of each supply chain financial user and the user grouping rule, so as to obtain each corresponding crowd pack, and then send each crowd pack to the Doris tag library 33 for storage.

In order to further improve the efficiency, reliability and effectiveness of crowd-sourced query in the management of supply chain financial user tags, in the system for managing supply chain financial user tags provided in the embodiments of the present application, the interface service management module 12 in the user tag management platform 10 in the system for managing supply chain financial user tags is further configured to receive a target crowd-sourced query request, and send the target crowd-sourced query request to the background application subsystem 20.

Correspondingly, the background application subsystem 20 is further configured to query, based on a preset unified service interface platform, crowd-sourced query result data corresponding to the target crowd-sourced query request from the Doris tag library 33 when or after receiving the target crowd-sourced query request, and send the crowd-sourced query result data to the interface service management module 12 for viewing by a corresponding querying user.

In order to further improve the convenience, reliability and effectiveness of task monitoring in the supply chain financial user tag management, in the supply chain financial user tag management system provided by the embodiment of the present application, referring to fig. 2, the user tag management platform 10 in the supply chain financial user tag management system further specifically includes the following contents:

The task monitoring management module 15.

The task monitoring management module 15 is configured to invoke, via the background application subsystem 20, execution status data of the current tag task execution of the Spark computing module 32 based on a preset plug-in, and display the execution status data.

In order to further improve the operation stability and data reliability of the supply chain financial user tag management system, referring to fig. 2, in the supply chain financial user tag management system provided by the embodiment of the present application, the user tag management platform 10 in the supply chain financial user tag management system further specifically includes the following contents:

the anomaly early warning management module 16.

The abnormality early warning management module 16 is in communication connection with the task monitoring management module 15, so as to send out corresponding abnormality early warning prompt information when the execution state data acquired by the task monitoring management module 15 is abnormal.

In order to further improve the operation stability and the reliability of user authority management of the supply chain financial user tag management system, referring to fig. 2, in the supply chain financial user tag management system provided by the embodiment of the present application, the user tag management platform 10 in the supply chain financial user tag management system further specifically includes the following contents:

The user rights management module 17.

The user rights management module 17 is configured to receive user information of a management user and/or a query user, and determine, based on pre-stored user rights comparison data, a user right corresponding to the user information, so as to perform rights management on the management user and/or the query user.

In order to further improve the intuitiveness and viewing convenience of the data output of the supply chain financial user tag management system, in the supply chain financial user tag management system provided in the embodiment of the present application, referring to fig. 3 and fig. 4, the background application subsystem 20 in the supply chain financial user tag management system is further in communication connection with an external application 40 outside the supply chain financial user tag management system.

Correspondingly, the background application subsystem 20 is further configured to extract the respective user tag of each of the supply chain financial users from the Doris tag library 33, generate visual data corresponding to the respective user tag of each of the supply chain financial users, and then send the visual data to the external application 40 and/or the user tag management platform 10.

In order to further improve the operation stability and the application reliability of the supply chain financial user tag management system, in the supply chain financial user tag management system provided by the embodiment of the application, the background application subsystem 20 is constructed and obtained in advance based on a Spring Boot frame, and the user tag management platform 10 is constructed and obtained in advance based on a vue.js frame.

Based on this, the embodiment of the present application is based on a Spring Boot framework, a vue.js technology, a relational database 31 such as MySQL, a big data computing engine Spark, and a Doris architecture, and is used for classifying, computing, and grouping tags, and the core technology is applied to big data computing engines Spark and Doris.

In order to further explain the above scheme, the present application further provides a specific application example of a supply chain financial user tag management system, referring to fig. 5, where the supply chain financial user tag management system uses a Spring Boot framework, a vue.js technology, a relational database such as MySQL, a big data computing engine Spark, and a Doris architecture to implement data extraction, computation, storage, and application of a user tag, so as to implement large-scale data processing and efficient execution of complex computing tasks. Through front-end and back-end separation and combination of a big data computing engine and a distributed storage architecture, the system has good expandability, performance and development efficiency, and the basic principle can be summarized as follows:

(1) Front-rear end separation: the front end and the rear end are separated, the front end uses Vue. Js technology to be responsible for the display and user interaction of a user interface, and the rear end adopts a Spring Boot frame to provide interface service and business logic processing.

(2) And (3) data storage: metadata of the user tag is stored in a MySQL database, and storage and management of the user tag data are achieved by defining a proper data model and a table structure.

(3) Tag calculation and generation: with Spark as the big data calculation engine, the system can perform efficient distributed calculation on a large-scale data set. And processing and converting the original data by writing a Spark task, and calculating out label information of the user according to a predefined label rule and algorithm.

(4) And (3) label storage: the calculated user tag information can be stored in a Doris architecture, and the Doris is a distributed column type storage engine, so that large-scale data can be stored and retrieved quickly. Through reasonable design table structure, realize the high-efficient storage and the inquiry to user's label data.

(5) Label application and presentation: the system can realize personalized recommendation, accurate marketing and other functions according to the label information of the user. By calling the back end interface, the front end can display the label information of the user and provide relevant functions and services.

(6) Task management and monitoring: the system provides a background management interface for managing configuration, maintenance and monitoring of the user tags. The label rule can be dynamically adjusted and updated, the running state of the task is monitored, and the problems are found and solved in time.

The execution flow of the user tag management platform 10 in the supply chain financial user tag management system is shown in fig. 6, for example.

Specifically, the execution content represented by each numerical identifier in fig. 6 is as follows:

1. The execution content indicated by the identifier "①" includes: the user label management module of the user label management platform defines the label code, label name, data type of label value, label classification and other information of the 1-4 grade label.

2. The execution content indicated by the identifier "②" includes:

(1) The label task is added to three-level labels defined in a user label management module of the user label management platform, wherein specific adding information comprises label task names, label task execution time, task levels, task execution parameters, label matching rules and label execution modes (SparkSql/Spark Jar), if the label task execution mode is SparkSql, label SQL needs to be added, and if the label task execution mode is Spark Jar, a label task main class needs to be added.

(2) And adding a flow task in the user tag task scheduling management module, wherein the main adding content comprises setting the task state as enabled, the task name, the execution mode as Spark Jar, the task execution time, the task main class, the task level and the task parameter.

3. The execution content indicated by the identifier "③" includes:

and uploading the universal task program Jar package to the HDFS file system for storage in the user tag task scheduling management module.

4. The execution content indicated by the identifier "④" includes:

And manually scheduling tasks to start execution at the user tag task scheduling management module, or automatically triggering task execution according to the timing task time set by the system.

5. The execution content indicated by the identifier "⑤" includes:

after the task is started, the background system automatically sends the relevant information of the task in the Json format to a remote task submitter according to the program to produce and submit a task command.

6. The execution content indicated by the identifier "⑥" includes:

the Spark task is initiated by SPARK LANCHER, submitted to the big data cluster Yarn.

7. The execution content indicated by the identifier "⑦" includes:

Spark jobs are produced, and tasks are run in Spark-Yarn clusters.

8. The execution content indicated by the identifier "⑧" includes:

the started task can query SQL sentences according to the data in the task information, and query the data from an offline database Doris.

9. The execution content indicated by the identifier "⑨" includes:

after the data is queried by the started task, the data is written into a user tag database Doris after data conversion and cleaning.

10. The execution content indicated by the identifier "⑩" includes:

in the task execution process, the remote task presenter updates the task state in real time, displays the updated task state in a task state column of the label task monitoring module, and returns a final state value after the task execution succeeds or fails. .

The supply chain financial user tag management system adopts various technologies and frameworks, such as Spring Boot framework, vue.js technology, mySql database, spark computing engine, doris storage engine, quartz task scheduling framework and the like, and the functions of user tag acquisition, calculation, storage and application, system monitoring, management, authority control and the like are realized through integration and application of the technologies. The core function model of the part is specifically described as follows:

(1) User tag management module 11

The user tag management module 11 is mainly responsible for definition, maintenance and task definition of user tags. The specific implementation mode of the module is as follows:

tag rule definition: by configuring the tag rule, the conditions, algorithm, output format, and the like of tag calculation are specified. The tag rule includes a tag name, a tag type, a tag calculation mode, and the like.

Spark task writing: and using Spark as a big data calculation engine, writing a Spark task, processing and converting the original data, and calculating out label information of the user according to a predefined label rule and algorithm.

Doris stores: and storing the calculated user tag information in a Doris architecture, and designing a proper table structure and index to support quick data storage and retrieval.

The association relationship between the user tag management module 11 and other modules is as follows:

the label task scheduling management module 13 may schedule Spark tasks defined by the user label management module 11, so as to ensure the order and efficiency of label calculation.

The task monitoring management module 15 can monitor the execution condition and state of the Spark task of the user tag management module 11, discover and solve problems in time, and ensure the normal operation of the task.

The user grouping management module 14 may group users using the tag information calculated by the user tag management module 11.

(2) Label task scheduling management module 13

The label task scheduling management module 13 is mainly responsible for scheduling and managing user label computing tasks and defining flow tasks. The specific implementation mode of the module is as follows:

Spark flow task writing: using Spark as big data calculation engine, writing Spark task, performing label wide table aggregation on label data calculated by user label management module 11, and writing conversion format of wide table data into Bitmap table type.

Quartz scheduling framework: and using the Quartz as a task scheduling framework to realize the scheduling and control of the label task, and setting the execution time, frequency, priority and the like of the task.

Spark task monitoring: the executing condition and the state of the Spark task are monitored in real time, the problems are found and solved in time, and the normal operation of the task is ensured.

The association relationship between the label task scheduling management module 13 and other modules is as follows:

user tag management module 11: the label task scheduling management module 13 may schedule Spark tasks defined by the user label management module 11, so as to ensure the order and efficiency of label calculation.

Task monitoring management module 15: the task monitoring management module 15 can monitor the execution condition and state of the Spark task of the label task scheduling management module 13, discover and solve problems in time, and ensure the normal operation of the task.

(3) Task monitoring management module 15

The task monitoring management module 15 is mainly responsible for monitoring and managing the running state of the task. The specific implementation mode of the module is as follows:

Acquiring a task execution state: the submitted task monitors the change of the state of the task in real time through a task monitoring management page, and a callback function is written to acquire the real-time change of the state through SparkLauncher plug-ins. SparkLauncher is a component of the Spring Boot rich ecosystem Spring Cloud, also part of APACHE SPARK, which provides a convenient way to launch, run and kill Spark applications in a separate Java process.

The association relationship between the task monitoring management module 15 and other modules is as follows:

The task monitoring management module 15 can monitor the execution condition and state of the Spark task of the user tag management module 11, discover and solve problems in time, and ensure the normal operation of the task.

The task monitoring management module 15 can monitor the execution condition and state of the Spark task of the label task scheduling management module 13, discover and solve problems in time, and ensure the normal operation of the task.

(4) User grouping management module 14

The user grouping management module 14 is mainly responsible for grouping users, and dividing the users into different groups according to a certain standard. The specific implementation mode of the module is as follows:

grouping rule definition: by configuring the grouping rules, the grouping conditions and algorithms are specified.

Doris grouping calculation: and according to the configured grouping rules, using Doris to calculate the cross-correlation difference of the label data of the Bitmap table, and according to the predefined grouping rules and algorithms, dividing the users into different groups.

Doris stores: and storing the calculated user grouping information in a Doris database.

The association relationship between the user grouping management module 14 and other modules is as follows:

the user grouping management module 14 may group users using the tag data calculated by the user tag management module 11.

(5) User rights management module 17

The user rights management module 17 is mainly responsible for rights management of the user. The specific implementation mode of the module is as follows:

user registration and login: the registration and login functions of the user are realized, and the personal information and data security of the user are protected.

And (3) authority control: and controlling the access and operation authority of the user to the system resources and data according to the user roles and authorities, and ensuring the safety and stability of the system.

Therefore, the supply chain financial user label management system provided by the application example of the application provides an implementation method and architecture design of the supply chain financial user label management system, and the supply chain financial user label management system implemented by adopting spark+doris architecture can optimize the calculation efficiency and the difficult maintenance problem of the traditional script mode. Through interface operation, the efficiency of developers can be improved, and the operation and maintenance cost is reduced. Specifically, compared with the existing label management mode, the label management system for the supply chain financial users provided by the application example has the following beneficial effects:

(1) Efficient processing of large-scale data

Conventional script implementations are typically stand-alone or simple clustered environments with limited computing and storage capabilities when processing large-scale data. The user tag management system based on Spark and Doris architecture has distributed computing and data storage technology, and can make full use of cluster computing resources to rapidly process large-scale data sets. The Spark framework supports memory calculation and disk IO processing, can buffer data in the memory, reduces disk IO times, and improves calculation efficiency.

(2) Supporting parallel computing and resource management

The user label management system of Spark and Doris architecture supports parallel computing, automatically manages the parallelism of tasks and resource allocation in a cluster, and improves the utilization efficiency of computing resources. Meanwhile, the resource manager is integrated, so that the allocation and release of the computing resources can be effectively managed, and the resource waste and bottleneck problems are avoided.

(3) High fault tolerance and reliability

The user label management system based on Spark and Doris architecture has good fault tolerance, and if a certain node or task fails, the system can automatically recover and retry, so that the reliability of the computing task is ensured. Meanwhile, doris is used as a distributed storage system, redundant backup and automatic fault transfer of data are supported, and safety and reliability of the data are ensured.

(4) High scalability

The user tag management system of Spark and Doris architecture supports lateral expansion, and more computing nodes and storage nodes can be added according to requirements so as to cope with challenges of increasing data volume and changing computing requirements. The system can dynamically adjust the resource allocation according to the actual situation, and ensures the scalability of the processing capacity.

(5) Better code maintainability

When Spark is used for writing tasks, a structured programming model can be adopted to divide complex calculation logic into a plurality of stages, so that the readability and maintainability of codes are improved. Meanwhile, spark provides rich APIs and libraries, code multiplexing and modularized development can be conveniently carried out, and redundant and repeated codes are reduced.

(6) Supporting distributed data storage and computing

Doris, as a distributed columnar storage system, may provide efficient data storage and query capabilities. The method supports technologies such as data compression and indexing, can quickly search and query the user tag data stored in the method, and improves the data access efficiency.

(7) Flexible and convenient monitoring and scheduling

Spark provides rich monitoring and scheduling tools, and can monitor the execution state, performance index and the like of tasks in real time, thereby facilitating system management and optimization. Meanwhile, the execution sequence and time of the tasks can be automatically scheduled by using the scheduler, so that the flexibility and the automation degree of task execution are improved.

It should be noted that, the portion of the background application subsystem 20 that performs data processing may be executed in a background server, or may be directly implemented as a server; the data processing part of the user tag management platform 10 may be implemented in the client device, or may be implemented directly in the client device. Specifically, the selection may be made according to the processing capability of the client device, and restrictions of the use scenario of the user. The application is not limited in this regard. If all operations of the user tag management platform 10 are completed in the client device, the client device may further include a processor for performing specific processing of the user tag management platform 10.

The client device may have a communication module (i.e. a communication unit) and may be connected to a remote server in a communication manner, so as to implement data transmission with the server. The server may include a server on the side of the task scheduling center, and in other implementations may include a server of an intermediate platform, such as a server of a third party server platform having a communication link with the task scheduling center server. The server may include a single computer device, a server cluster formed by a plurality of servers, or a server structure of a distributed device.

Any suitable network protocol may be used between the server and the client device, including those not yet developed on the filing date of the present application. The network protocols may include, for example, TCP/IP protocol, UDP/IP protocol, HTTP protocol, HTTPS protocol, etc. Of course, the network protocol may also include, for example, RPC protocol (Remote Procedure Call Protocol ), REST protocol (Representational STATE TRANSFER) or the like used above the above-described protocol.

The present application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter, where the processor is configured to execute the data processing content executed by the user tag management platform 10 or the data processing content executed by the background application subsystem 20 in the supply chain financial user tag management system mentioned in the foregoing embodiment, and the processor and the memory may be connected by a bus or other manners, for example, through a bus connection. The receiver may be connected to the processor, memory, by wire or wirelessly.

The processor may be a central processing unit (Central Processing Unit, CPU). The Processor may also be other general purpose processors, digital Signal Processors (DSP), application SPECIFIC INTEGRATED Circuits (ASIC), field-Programmable gate arrays (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above.

The memory, as a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to data processing content executed by the user tag management platform 10 or data processing content executed by the background application subsystem 20 in the supply chain financial user tag management system according to an embodiment of the present application. The processor executes the various functional applications and data processing of the processor by running non-transitory software programs, instructions and modules stored in memory, i.e., implementing the data processing content executed by the user tag management platform 10 or the data processing content executed by the background application subsystem 20 in the supply chain financial user tag management system in the system embodiments described above.

The memory may include a memory program area and a memory data area, wherein the memory program area may store an operating system, at least one application program required for a function; the storage data area may store data created by the processor, etc. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory may optionally include memory located remotely from the processor, the remote memory being connectable to the processor through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The one or more modules are stored in the memory and, when executed by the processor, perform the data processing performed by the user tag management platform 10 in the supply chain financial user tag management system or the data processing performed by the background application subsystem 20 in the embodiment.

In some embodiments of the present application, a user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter, the processor, the memory, the receiver, and the transmitter may be connected by a bus system, the memory being configured to store computer instructions, the processor being configured to execute the computer instructions stored in the memory to control the transceiver unit to transmit and receive signals.

As an implementation manner, the functions of the receiver and the transmitter in the present application may be considered to be implemented by a transceiver circuit or a dedicated chip for transceiver, and the processor may be considered to be implemented by a dedicated processing chip, a processing circuit or a general-purpose chip.

As another implementation manner, a manner of using a general-purpose computer may be considered to implement the server provided by the embodiment of the present application. I.e. program code for implementing the functions of the processor, the receiver and the transmitter are stored in the memory, and the general purpose processor implements the functions of the processor, the receiver and the transmitter by executing the code in the memory.

Embodiments of the present application also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the data processing content executed by the user tag management platform 10 or the data processing content executed by the background application subsystem 20 in the foregoing supply chain financial user tag management system. The computer readable storage medium may be a tangible storage medium such as Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disk, a removable memory disk, a CD-ROM, or any other form of storage medium known in the art.

Those of ordinary skill in the art will appreciate that the various illustrative components, systems, and systems described in connection with the embodiments disclosed herein can be implemented as hardware, software, or a combination of both. The particular implementation is hardware or software dependent on the specific application of the solution and the design constraints. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine readable medium or transmitted over transmission media or communication links by a data signal carried in a carrier wave.

It should be understood that the application is not limited to the particular arrangements and instrumentality described above and shown in the drawings. A detailed description of known systems is omitted here for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. The system processes of the present application are not limited to the specific steps described and illustrated, but various changes, modifications and additions, or the order between steps may be made by those skilled in the art after appreciating the spirit of the present application.

In this disclosure, features that are described and/or illustrated with respect to one embodiment may be used in the same way or in a similar way in one or more other embodiments and/or in combination with or instead of the features of the other embodiments.

The above description is only of the preferred embodiments of the present application and is not intended to limit the present application, and various modifications and variations can be made to the embodiments of the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims (10)

1. A supply chain financial user tag management system, comprising: the system comprises a background application subsystem, a user tag management platform and a big data cluster subsystem which are respectively in communication connection with the background application subsystem;

The big data cluster subsystem is also provided with a relational database, a Spark computing module and a Doris tag library;

The user tag management platform is used for acquiring tag definition data, and the tag definition data comprises: corresponding relations among various user label types, user label setting rules and label tasks of the supply chain financial users;

The background application subsystem is used for sending the label definition data received from the user label management platform to the relational database as metadata for storage, and sending the label task to the Spark computing module;

The Spark computing module is used for retrieving each user tag type and user tag setting rule corresponding to the tag task from the relational database when or after the tag task is received, acquiring the raw data of the supply chain financial user corresponding to the tag task from an offline data warehouse, further determining each user tag of each supply chain financial user corresponding to the raw data of the supply chain financial user according to each user tag type and user tag setting rule corresponding to the tag task, and sending each user tag of each supply chain financial user to the Doris tag library;

the Doris tag library is used for storing received user tags of the supply chain financial users.

2. The supply chain financial user tag management system of claim 1, wherein the user tag management platform comprises: a user tag management module and an interface service management module;

the user tag management module is used for receiving the tag definition data and sending the tag definition data to the background application subsystem;

The interface service management module is used for receiving a target user tag query request and sending the target user tag query request to the background application subsystem;

Correspondingly, the background application subsystem is further used for inquiring user tag inquiry result data of the target supply chain financial user corresponding to the target user tag inquiry request from the Doris tag library based on a preset unified service interface platform when or after the target user tag inquiry request is received, and sending the user tag inquiry result data to the interface service management module for the corresponding inquiry user to check.

3. The supply chain financial user tag management system of claim 2, wherein the user tag management platform further comprises: a label task scheduling management module;

The tag task scheduling management module is used for retrieving the tag definition data from the background application subsystem, setting Spark flow tasks taking Quartz as task scheduling frames according to the tag definition data, and then sending scheduling and monitoring instructions aiming at the tag tasks to the background application subsystem by the Spark flow tasks so that the background application subsystem sends the tag tasks to the Spark calculation module according to execution time, frequency and priority appointed by the scheduling and monitoring instructions of the tag tasks, and the Spark calculation module is used for generating user tags of each supply chain financial user corresponding to the tag tasks.

4. The supply chain financial user tag management system of claim 2, wherein the user tag management platform further comprises: a user grouping management module;

the user grouping management module is used for receiving the user grouping rule and sending the user grouping rule to the background application subsystem;

Correspondingly, the background application subsystem is further configured to retrieve respective user tags of the supply chain financial users from the Doris tag library, perform grouping processing on the supply chain financial users according to the respective user tags of the supply chain financial users and the user grouping rules, so as to obtain corresponding crowd packages, and then send the crowd packages to the Doris tag library for storage.

5. The supply chain financial user tag management system of claim 4, wherein the interface service management module is further configured to receive a target crowd-sourced query request and send the target crowd-sourced query request to the background application subsystem;

Correspondingly, the background application subsystem is further used for inquiring crowd-sourced inquiry result data corresponding to the target crowd-sourced inquiry request from the Doris tag library based on a preset unified service interface platform when or after the target crowd-sourced inquiry request is received, and sending the crowd-sourced inquiry result data to the interface service management module for the corresponding inquiry user to check.

6. The supply chain financial user tag management system of claim 2, wherein the user tag management platform further comprises: a task monitoring management module;

The task monitoring management module is used for calling the execution state data of the current tag task execution of the Spark computing module through the background application subsystem based on a preset plug-in, and displaying the execution state data.

7. The supply chain financial user tag management system of claim 6, wherein the user tag management platform further comprises: an abnormality early warning management module;

The abnormal early warning management module is in communication connection with the task monitoring management module, so that corresponding abnormal early warning prompt information is sent out when the execution state data acquired by the task monitoring management module is abnormal.

8. The supply chain financial user tag management system of claim 2, wherein the user tag management platform further comprises: a user rights management module;

The user authority management module is used for receiving user information of a management user and/or a query user, and determining user authority corresponding to the user information based on prestored user authority comparison data so as to carry out authority management on the management user and/or the query user.

9. The supply chain financial user tag management system of any one of claims 1 to 8, wherein the background application subsystem is further communicatively coupled to an external application external to the supply chain financial user tag management system;

correspondingly, the background application subsystem is further used for extracting the user labels of the supply chain financial users from the Doris label library, generating visual data corresponding to the user labels of the supply chain financial users, and then sending the visual data to the external application and/or the user label management platform.

10. The supply chain financial user tag management system of any one of claims 1 to 8, wherein the background application subsystem is previously built based on a Spring Boot framework, and the user tag management platform is previously built based on a vue.