CN111523808B

CN111523808B - Model centralized management method, system, equipment and storage medium

Info

Publication number: CN111523808B
Application number: CN202010333117.7A
Authority: CN
Inventors: 沈丽忠; 李婉华; 詹炜华
Original assignee: China Construction Bank Corp
Current assignee: China Construction Bank Corp
Priority date: 2020-04-24
Filing date: 2020-04-24
Publication date: 2022-12-16
Anticipated expiration: 2040-04-24
Also published as: CN111523808A

Abstract

The invention discloses a model centralized management method, a system, equipment and a storage medium, wherein the method mainly comprises the following steps: registering all models needing centralized management in an enterprise to the same model management platform; and determining a matched available model in the model management platform according to the authority corresponding to different platform roles so as to enable the platform roles to share and reuse the model assets based on the available model. By adopting the invention, all models are registered to the same model management platform, and the models are shared and reused in the platform according to the authorities of different roles, so that the centralized model management can be realized, and the problems of model dispersion, incapability of sharing and reusing across parts and the like in the centralized management are solved.

Description

Model centralized management method, system, equipment and storage medium

Technical Field

The present invention relates to the field of model management technologies, and in particular, to a method, a system, a device, and a storage medium for centralized model management.

Background

In various financial institutions of countries in the world, models are widely developed and applied to scenes such as marketing, wind control and decision making, and become indispensable important tools in various financial services including credit services. However, there is also a risk due to errors or improper use of the model itself, which is the risk of the model. For example, a crash from Long Term Capital Management (LTCM) in 1998, the U.S. sublicense crisis in 2007, the Morgan chase university (JP Morgan chase) "london whale" event 2012, etc. are all associated with model risks, indicating that model risks have had enormous destructiveness as only one operational risk, and the absence of a model risk management system for an institution or even a country would pose an immeasurable systematic risk of destructiveness for the institution or country.

Model risk management is a long-term, systematic, complex and detailed work, and in the absence of efficient methods and tools, a large amount of human resources must be occupied, and the timeliness and quality of the work may still be difficult to guarantee.

In the existing model management, with the diversification of service lines and the complexity of service scene types, the number of models in a mechanism and the work tasks of the models are rapidly increased. Generally, model work can be independently carried out between each service line in the organization, so that model assets in the organization are dispersed, historical information related to the model cannot be traced back, internal audit requirements cannot be met, or compliance risk potential hazards caused by the fact that effective model documents cannot be provided exist; the state of the model cannot be known in time, so that the working state of the model cannot be effectively tracked, and delay is caused, or the operation loss caused by the performance loss of the model cannot be known in time; the model assets can not be shared and reused across departments, part of model work is repeated, and human resources are consumed.

Disclosure of Invention

The embodiment of the invention provides a method, a system, equipment and a storage medium for centralized management of a model, which can solve the relevant problems in the centralized management of the model by constructing an enterprise-level unified model management platform.

A first aspect of an embodiment of the present invention provides a method for centralized model management, where the method includes:

registering all models needing centralized management in an enterprise to the same model management platform;

and determining a matched available model in the model management platform according to the authority corresponding to different platform roles so as to enable the platform roles to share and reuse model assets based on the available model.

A second aspect of the embodiments of the present invention provides a model centralized management system, which may include:

the system comprises a registration database building module, a model management platform and a database management module, wherein the registration database building module is used for registering all models needing centralized management in an enterprise to the same model management platform;

and the asset sharing and multiplexing module is used for determining a matched available model in the model management platform according to the authority corresponding to different platform roles so that the platform roles can share and multiplex the model assets based on the available model.

A third aspect of embodiments of the present invention provides a computer storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform the following steps:

and determining a matched available model in the model management platform according to the authority corresponding to different platform roles, so that the platform roles can share and reuse the model assets based on the available model.

A fourth aspect of embodiments of the present invention provides an apparatus, which may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the steps of:

The invention has the beneficial effects that:

by positioning at an enterprise level, assisting an organization to establish an enterprise-level model asset library, carrying out centralized management on models applied to different service scenes, and registering all models and related documents thereof on the same platform;

according to the given platform authority, a user can easily find needed model assets on the platform, the model assets are shared and reused in the mechanism, and the intelligent development is promoted by mutually referencing the model assets.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a schematic flow chart of a method for centralized management of models according to an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of an enterprise-level model management system according to an embodiment of the present invention;

FIG. 3 is a block diagram of another enterprise-level model management system provided by an embodiment of the present invention;

FIG. 4 is an architecture diagram of an enterprise-level model management system provided by an embodiment of the present invention;

FIG. 5 is a diagram of dependency relationships between model forms, model workflows, and model registration functions provided by an embodiment of the present invention;

FIG. 6 is an architecture diagram of a process engine provided by an embodiment of the present invention;

FIG. 7 is an architecture diagram of a model evaluation service provided by an embodiment of the present invention;

FIG. 8 is an architecture diagram of a data reflow service provided by an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of a model centralized management system according to an embodiment of the present invention;

FIG. 10 is a schematic structural diagram of a registration library building module provided in the embodiment of the present invention;

FIG. 11 is a block diagram of a workflow creation module according to an embodiment of the present invention;

fig. 12 is a schematic structural diagram of an apparatus according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The terms "including" and "having," and any variations thereof, in the description and claims of this invention and the above-described drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may alternatively include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Furthermore, the terms "mounted," "disposed," "provided," "connected," and "sleeved" are to be construed broadly. For example, it may be a fixed connection, a removable connection, or a unitary construction; can be a mechanical connection, or an electrical connection; may be directly connected, or indirectly connected through intervening media, or may be in internal communication between two devices, elements or components. The specific meanings of the above terms in the present invention can be understood by those of ordinary skill in the art according to specific situations.

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

The device related to the embodiment of the invention can be a mainframe computer, a PC, and other computer processing devices with data processing capability.

As shown in fig. 1, in a first embodiment of the present application, a model centralized management method at least includes the following steps:

and S101, registering all models needing centralized management in an enterprise to the same model management platform.

In specific implementation, the system firstly needs to centralize all models in an enterprise to a system platform for management, breaks barriers of data and models between business departments, makes asset sharing and multiplexing possible, and then can perform enterprise-level management. For any model within an enterprise, the model and related documents need to be registered to the platform. Optionally, the enterprise may also transfer the historical models and related files to the unified management platform by means of an interface, so as to perform unified management on all models in one enterprise.

The following illustrates the benefits of building an enterprise-level model asset library for model management in a new business, i.e., cold-start scenario: in a scene of cold start of a service, a model corresponding to a new service is often developed without a method because of no data, and for the same link in a credit service, a model which is already used in other services can be considered to be multiplexed into the new service, which may not be perfect, but how many models of different services applying the same link endow the new service with certain wind control capability. Specifically, a worker related to the new service may obtain a corresponding right given by the system, the worker may log in the system to check whether a model of another service is applicable and is multiplexed into a decision flow of the new service, the worker may copy and deploy some manual analysis components related to an available model into a production environment of the new service, and the worker also needs to create a new model definition, a new model requirement, and the like in the system.

In a specific implementation, the process of model registration may include the following steps:

1) Creating a model definition: model name, model description, database and data sheet corresponding to the model.

2) Defining model metadata: model name, model description, business to which the model belongs, business scenario in which the model is used, model type, model output type, model affiliation person, sample window in model development, model release date, and the like.

3) And connecting the external database corresponding to the model and the data form of the model.

4) Variables used by the model are selected for the data form in the platform to be used for generating a model verification report, and the variables are endowed with a report role, such as the following characteristics for defining a variable as the model use: identifying the variables used by the model and assigning roles to the model features, identifying other variables needed for reporting and assigning related roles, reporting roles having a model call date role, a model score role, a sample weight role, a model target role, and the like.

5) Configuring a model, uploading a model file, wherein the actual file format of the model is pmml or pkl, the model file is called when a report is generated, and the calling aims at identifying the model when the platform generates the report; defining data windows and benchmarks, defining model prediction objectives and assigning reporting roles-objectives, defining score distributions and the cut groups to which the models belong.

6) Optionally, the model-related supporting files may be uploaded simultaneously.

7) Optionally, model-related comments may be attached.

In an alternative embodiment, the system may monitor within the model dashboard the model operating state, performance metrics, and health of the selected model over its lifecycle.

In specific implementation, after model registration and configuration of each model are completed, all models are already centralized on a system platform, and at this time, monitoring of the performance of all models, that is, the working state and performance index of each model, can be started. In the aspect of centralized monitoring, the system mainly reflects the working state of the model, the performance index of the model and the health state of the model in a centralized manner in a mode of a model instrument board.

It should be noted that the dashboard in the system can display the current status of all registered models, and the user can perform model screening operation in the dashboard to locate the model requiring further operation, i.e. the selected model. Clicking on any model within the dashboard may navigate to a verification results page associated with the model, which may typically be displayed in the form of a sector statistical chart, bar statistical chart, or table. The complete report running behind the verification result (usually a metric) can be downloaded from the system platform, providing the pdf,. Xlsx,. Html file format.

Optionally, the expression mode of the model state variables and the verification results displayed on the model dashboard may be customized and adjusted according to the user's requirements. Generally, the model dashboard displays the name of the model, the cutting group where the model is located, the working environment to which the model belongs, the alarm level and the number of alarm rule triggers of the model, the general health profile of the model, the working period of the model, the number of days since the last verification of the model, and other information.

Optionally, the system platform may generate an alarm signal/event according to the defined alarm rule and the verification result index and feed back the alarm signal/event to the model dashboard. The user can view the single alarm or all alarms of the model to track the alarm reason and arrange for further work. The user can modify the current alarm rules and can add more alarm rules.

Optionally, the person to whom the model belongs may change the health status of the model and add comments according to the verification result. The model health status levels can be classified as healthy, attentive-mild, attentive-moderate, attentive-urgent, requiring immediate treatment, etc.

Alternatively, the relevant person of each model may add relevant comments to record observations of the model while historical activities of the model may be viewed.

S102, determining a matched available model in a model management platform according to the authority corresponding to different platform roles, so that the platform roles can share and reuse model assets based on the available model.

It should be noted that the platform roles in the system may include at least the model owner, workflow author, model approver, and system administrator. Wherein:

the person to which the model belongs: the owner of the model, a specific initiator of the new model requirement, and the owner of the model can have all system role authorities except a system administrator, and can delete, modify, and the like the model with the viewing authority.

Workflow author: the workflow author creates and releases the model workflow in the platform system, and defines the subtasks in the workflow according to the model work tasks to assign the subtasks in the workflow to the responsible persons, and the workflow author and the person to which the model belongs can be the same person.

The model authors: the model author is responsible for completing subtasks in the model workflow, the model author can be further subdivided into data analysis personnel, model development personnel, model verification personnel and model deployment personnel according to different internal positions of the using mechanism, and other relevant personnel of the model work except for model examination and approval personnel, and the model author is responsible for creating model definition, configuring model metadata, relevant model variables and variable roles, developing the model, creating a model verification plan, composing alarm rules and the like.

Model approver: different from a model author, a model approver is generally not specifically responsible for actual operation of model work, and the model approver is responsible for an approval subtask in a model workflow and is responsible for approving output of the model author and completing the approval subtask.

A system administrator: the system administrator is responsible for authority management, configuring an external database and a data table corresponding to each model, deploying issued workflows, and configuring a custom model instrument board and a custom model verification report.

Self-defining roles: according to the working requirements of the model, the organization can create a self-defined role through a system administrator.

When the assets are shared and reused, different roles can share data or extract data according to respective platform authorities, and authority verification is needed at the same time, so that data confusion caused by the assets are shared and reused, and the stability of the platform is not affected.

In the second embodiment of the present application, the system may create a general model workflow based on the work requirement or the model work task of any model, so that all the model works can follow a relatively consistent flow, each link of the workflow is assigned to a specific responsible person, and according to the difference of the links, the work result to be uploaded is defined, and the approval links are configured in the workflow according to the needs, thereby ensuring the work quality of each work link in a single model workflow, and the requirements of the model work quality are unified between the workflow and the workflow. In addition, according to a binary or continuous model target, a corresponding model standardized report is embedded in the model workflow, a user can customize a report generation period (quarterly, monthly and weekly), a data time range (the first 3 months, the first 6 months or other backtracking time ranges), and the platform automatically generates the report.

In specific implementation, any task is put into a universal model workflow, the model workflow covers sub-links of a plurality of model works, and for different model task requirements, a workflow author can perform custom modification on the basis of the universal model workflow, so that all model works are ensured to follow certain normative, and flexibility is provided according to respective model work requirements. When the workflow is released and activated, the workflow automatically follows up the working progress and automatically reminds a person in charge of the sub-link to complete the task.

In an optional embodiment, for a single work task or a combination of multiple work tasks such as model development, model verification, model maintenance, model deployment, model optimization, and the like, a corresponding workflow definition corresponding work range and a required sub-work link can be established. The workflow author defines the subtask segments needed in the workflow and assigns the subtasks to specific principals or responsible teams. It can be understood that all model work users (including subtask link responsible persons or responsible teams) related to the workflow can see tasks to be completed in the respective systems, and the users can also see where task links responsible for the users are located in the whole model workflow, so that time is arranged in advance, and communication among link responsible persons is promoted.

In a preferred implementation, the step of creating a model workflow for a model work requirement or a model work task may include the following processes:

1) A workflow author creates a workflow and defines subtasks in the workflow, and when the workflow author creates the workflow, the workflow author needs to define the name of the workflow, the type of the workflow and a corresponding general workflow template; when defining the subtasks, the workflow author needs to be aware of the subtasks required by the model task and assign the subtasks to people when creating the workflow.

2) The workflow author issues the workflow, a system administrator deploys the workflow without errors, and the workflow is automatically advanced according to the defined task links and the responsible persons.

3) And the workflow sends a mail to a responsible person of the next subtask according to the progress, and related workers of the task receive the mail prompt and the system login link and obtain corresponding system permissions conforming to the workers at the same time, so that model lookup, modification and other model related work permissions can be carried out.

4) When the worker in charge of the subtask completes and uploads the document required by the task, the subtask is simply completed, and the workflow automatically advances to the next subtask and notifies the relevant person in charge. And (4) repeating the steps 3 and 4 by the workflow until all subtasks of the workflow are completed, and determining whether to close the workflow or not by a workflow author according to the work quality and the workflow progress and marking a completion label.

5) All users authorized to review the model or workflow may review all historical activities and associated records during the validation of the workflow.

6) If the workflow author modifies or updates the workflow during the validation of the workflow, the workflow is updated according to the update, the workflow needs to be reissued, a system administrator needs to be redeployed, and the workflow is restarted from the latest unfinished subtask and pushed to work.

In the embodiment, a set of enterprise-level management method and system based on a life cycle and utilizing configurable and automated workflows is established, so that risk and analysts of an organization can continuously develop a leading-edge model and a leading-edge strategy with competitive advantages and follow a designed workflow, and information, output and important files related to all models are unified into a library. By setting role authority, different personnel and departments can properly use and share information in the library, so that the requirements from the interior of the organization are met under the condition of saving most human resources. For the work scenario of model development, the workflow guides the user through a standardized flow of recorded actions and decisions. It avoids manual data entry by automatically obtaining segmentation criteria, test results, and other key information. In addition, the workflow can also coordinate cross-department model audit, timely inform related personnel and ensure that the next subtask is carried out after the current subtask is approved. The dashboard of the workflow may also provide a clear real-time status for the progress of all models. In response to regulatory and internal reporting or query requirements, the workflow works by importing all data from the model lifecycle into a centralized repository. On such a unified platform, which contains all the information about the model, the task of preparing the user for audits or writing reports for responding to queries becomes faster and more efficient. For example, they can check the most recent verification tests for a model very quickly, and can also easily peruse the entire process history, including each verification that the model is doing, and the results and subsequent operations in each instance.

In summary, the enterprise-level model management platform based on the automated and configurable workflow can realize centralized model management, so that workers can quickly know all histories of development (development data/model development/model verification), deployment (complexity/operation requirement/operation test of the model), and maintenance (verification/modification/iteration after model delivery) of each model. Meanwhile, the bank can be ensured to retain all the information related to the model and support the basis of the related decision, and the inquiry from the internal requirement of the bank and the external supervision is met to the maximum extent. In addition, besides the built-in workflow, the platform can simultaneously carry out the periodic automatic verification of the model and mark the expression of the model according to the change of the performance index of the model. This allows the bank to learn the state of the model more timely, to maintain the model in operation more easily, and to ensure that it is at a high performance level.

As shown in fig. 2, in a third embodiment of the present application, the role authority management in the enterprise-level model management architecture is mainly used for managing the authority of different roles in different business teams, where the roles have authority in the system platform, and each role can view interested data through the dashboard according to the respective matched authority, for example, centralized management can be performed through the dashboard 1, content in the model list can be viewed, verification and monitoring can be performed through the dashboard 2, and verification and monitoring results of a single or multiple models can be viewed. In addition, model manifests and validation and monitoring results of single or multiple models can be shared and multiplexed to different business teams.

In a specific implementation scenario, the permissions corresponding to different roles may complete different tasks, for example:

the person to which the model belongs: the person to which the model belongs enters a platform system to inquire the states of all models managed by the person at present, and specific historical activities or present model expression conditions of a certain model can enter the working environment of the model to inquire.

Workflow authors: the workflow author enters a workflow management page of the platform system to create a model workflow according to a new model task, and can also select a general workflow corresponding to the new model development for the new model development task.

The model authors: after the workflow is formally deployed and validated, the model author of the current first subtask receives task mail and system connections from the system as the workflow advances. For example, in a subtask-model development link, a model developer has the authority corresponding to the role of a model author in a system platform, and according to the requirements of the subtask, the model developer completes data cleaning to generate a data quality report, performs index derivation and feature screening to generate a feature report, performs model development and testing to generate a report, uploads the report to the subtask uniformly and submits the subtask.

Model approver: before the new model is on line, the model needs to be tested and verified, and model approvers check the normative and the rationality of the reports in the stages of model development, model development and model verification and give approval for whether the approval is passed. Only after approval by the model approver can the model work move on to the next link.

A system administrator: when a workflow author creates and uploads a workflow, a system administrator needs to deploy the workflow, and only when the workflow is deployed, the workflow will actually take effect and be displayed on a dashboard. For the responsible person assigned by the subtask in the workflow, the system administrator gives the corresponding system role and the related authority to the responsible person.

In this embodiment, the content management information base may correspond to the enterprise-level model asset base in the above-described embodiment. The process of model registration in the content management information base is as follows: firstly, different business teams create respective model definitions; further defining model metadata which mainly comprises a business line, a business scene, a model type, an output type, a model attribution person, a sample window, a release date and the like; then connecting the corresponding database of the model and the data table, then configuring the model, and setting the verification definition. Wherein the defined model metadata and the created model definition may be embodied in a model manifest. When single or multiple models are verified and monitored, the database and the data table corresponding to the monitored model, the configuration of the model and the definition of configuration verification are mainly verified.

In the fourth embodiment of the present application, from the perspective of principle function, the enterprise and model management system is divided into four important modules, namely, a model management platform, a workflow management system, and a model verification/evaluation system, and the working relationship in model risk management between different modules is shown in fig. 3:

relation line a: model component-model author-model working environment-content management system

For an existing model, a model author creates a model definition in a model working environment, uploads the relevant parts and relevant documents of the model, and the parts and relevant documents of the model are stored in a content management system of a model risk management platform.

Relation line b: model author-custom report-System administrator-System management-dashboard

Generally, after a template of a custom report is designed outside a model risk management platform by a model author, the template is uniformly managed and configured in the platform by a system administrator, after the configuration is completed, the model author can use the report to perform verification and monitoring work in a model working environment, and a verification and monitoring result can be fed back to a dashboard.

A relation line c; workflow author-workflow management-dashboard

The workflow author creates and maintains the workflow in the workflow management, when the workflow is released and deployed, the state of the workflow is gathered in the instrument panel, and besides the state, the workflow can urge the model author to enter the model working environment to complete tasks according to the work tasks and the responsible persons in the workflow.

Relation line d: the system administrator-system management/authority management system administrator is responsible for management of the daily system platform, maintenance of the platform environment, assistance of workflow authors in workflow deployment, daily platform log maintenance, and management of the role of the platform and the authority corresponding to the role.

Relation line e: model production environment (non-platform environment scope) -model reflow data (non-platform environment scope) -model management platform database (non-platform environment scope) -model verification/monitoring work-verification report and result-content management system-model working environment

The model is called in a production environment to generate model backflow data, the backflow data is processed by ETL and the like and flows back to a database externally connected with the model risk management platform, when a model author carries out model verification, monitoring and other work in the platform, a verification and monitoring engine in the platform calls data corresponding to the model in the externally connected database, a verification report and a result are generated by combining information such as model definition, variables and a verification report role in the platform, the verification report and the result are stored in a content management system, and the model author can check the verification report and the result in the model working environment.

Relation line f: when a model author completes corresponding work or tasks in the model work environment, the completed work or operation is stored in the content management system, the content management system and the workflow management system are communicated with each other, the workflow system identifies the next action, advances the workflow progress forward and returns the progress to the content management system.

In a fifth embodiment of the present application, the architecture of the enterprise-level model management system is shown in fig. 4, and includes a presentation layer, an access gateway layer, a system application service layer, and an infrastructure layer. Wherein:

and (3) displaying layer: the method is mainly used for login used by a user and visual display of a front-end operation interface of the system.

An access gateway layer:

nginx gateway: the network isolation and the unified management of the API are realized, the public network service is exposed to the outside, the intranet flow is shunted and forwarded internally, the API request protocol conversion is supported, the load of the request is distributed in a balanced manner, and the high-availability and unified security entry management of the system is improved.

2. And (3) access authentication: and performing access authentication on the API access. Preventing illegal request operations.

The system application service layer:

model asset management platform:

a) Model form: the most basic unit module in the model registration is composed of a plurality of fields, represents the classification combination of the fields, can be expanded according to the service customization and is quoted for the workflow model node.

b) Model workflow: and defining a model life cycle management process link for a carrier in the model registration process.

c) Model registration: registering model instance information, storing model registration details, and containing the content and associated files of the model.

The dependency relationship among the model form, the model workflow and the model registration function is shown in fig. 5: the model form can be customized and expanded according to the needs of a business scene, and different forms are combined with the workflow to finally form the whole process of model registration. Model nodes in the workflow need to refer to a certain model form. The model workflow is combined with the model form to generate a model registration processing process, and the change of nodes in the whole processing process of model registration needs to call the model workflow for processing. In the process of model registration processing, model data can be instantiated, the requirement information, development information, verification information and the like of the model are perfected, model registration is completed, and the instantiation data of the model is also completed.

And (4) reporting the data:

a) Operation and operation history records related to the model, such as the model (pmml, pdf, excel,. Pkl), the data report (. Xlsx,. Doc,. Pdf,. R), the verification report (. Doc,. Pdf, model code), etc., are respectively stored in the file object system OSS and the big data platform, and the timed task jobgenerates a BI report required by the service according to the service requirement and sends the BI report through a channel such as a mail.

b) The model operation platform can generate score and other dimensional data to be grounded on a database according to real-time calling conditions of production, the model management platform can acquire variable and data in the database from multiple dimensions in the database, and different dimensional reports are generated by combining dynamic script codes to form model verification or supervision reports.

A flow engine:

the core function of model management is to realize the process standardized management of model production, model registration, model examination and approval and the like, and the model management engine is an engine for driving the full life cycle management of the model, can support the configuration and the expandability of services, is in environment communication connection with a service model factory and a model operation platform, and supports the self-iteration of the model. The architecture of the flow engine is shown in fig. 6: the embedded activiviti open source flow platform develops a stateless springboot application, takes MySQL as a state storage layer, is deployed in an independent application cluster mode, and exposes a uniform domain name to the outside. The method conforms to BPMN standard and realizes a general process platform customized by companies. The flow node supports triggering an external HTTP interface, a restful interface related to the flow can be exposed to the outside, and the upper layer of the service can be directly called. The method comprises the steps of providing a web-version console, visually dragging and editing a process page, and enabling an editor to create or edit an activiti process file (the activiti is of a specific process file format, is essentially a ZIP package and comprises a standard BPMN process definition XML file and related resource files such as icons and the like), and a process instance query interface (API). And after the verification is finished, deploying the process file into the current activiti engine. The process nodes of the process engine can call the designated service interfaces in the service package, and the interfaces need to be authenticated to prevent illegal access and ensure the safety of service calling.

Model assessment and model validation/monitoring service:

model predictive assessment support predictive assessment of standard generic model files in model asset systems, with current designs supporting pmml, py model file type predictive assessment. The overall evaluation architecture is shown in fig. 7:

the front-end page sends the model code, the model data file or the designated model data source to the model evaluation service. The model evaluation service obtains model files and model report templates from the model asset management service according to the content uploaded by the front end, and obtains data from the existing configured data source if the model evaluation service points to the data source. And analyzing and operating the model file by the model evaluation service, and generating a model operation report after the operation is finished. The model evaluation service sends a model operation report to the model asset management service, and the model asset management service sends a notification to a specified mailbox. And the model evaluation service calls an acquisition interface of the big data cloud platform and issues the model operation data to the big data cloud platform.

The data backflow service mainly provides operation data backflow of a three-party model factory and a model operation platform, supports two modes of real-time synchronous backflow and offline batch file backflow, and provides a data source for model supervision, and mainly provides a data source for model development evaluation and backflow effect evaluation as shown in fig. 8. In fig. 8, modeling related data generated by the model development environment (i.e. model development modeling factory) and model operation environment (i.e. real-time data operation of model production, some configuration data and workflow data generated by the model management platform are transmitted to the message middleware kafka or similar data bus by way of data production. Data reflow service: and consuming service data generated by message middleware or similar data buses, performing format conversion and unified processing, and performing relevant preprocessing of service indexes. Data flows into the big data platform. And the big data platform runs a report task, performs spark, hive and other timing calculations, and visually checks and displays the result on the model management platform. The flow approval information and BI report forms related to the flow can reach the user terminal through a three-party information channel in real time.

Basic capability layer:

1) A big data platform: and the data storage and real-time spark stream calculation of a big data scene and the basic storage and calculation capacity such as hive are supported.

2) MySQL: and operations such as basic object fields such as model forms and model registration, storage such as process engine and model circulation states, business query and the like are supported.

3) ELK: the storage of log data is supported, and the viewing capacity is monitored and visualized based on the operation of some rules.

4) And (3) OSS: supporting model-dependent file object storage.

5) Kafka: and the asynchronous streaming of messages of data among systems and the loose coupling among systems are supported.

And (3) service monitoring:

and (4) model operation condition, online abnormal API call or abnormal model management flow operation, and real-time scripted calculation and alarm.

In a sixth embodiment of the present application, another model centralized management system is provided for the embodiment of the present invention. As shown in fig. 9, the model centralized management system 10 of the present embodiment may include: a registration library building module 101, an asset sharing multiplexing module 102, a dashboard monitoring module 103, a dashboard adjusting module 104, a workflow creation module 105, a workflow processing module 106, a review feedback module 107, and an update restart module 108. As shown in fig. 10, the registry library creating module 101 includes a model definition creating unit 1011, a metadata defining unit 1012, a data connecting unit 1013, a report processing unit 1014, a model configuring unit 1015, and an asset library forming unit 1016, and as shown in fig. 11, the workflow creating module 106 includes a subtask creating unit 1061, a workflow deploying unit 1062, a subtask processing unit 1063, and a subtask advancing unit 1064.

And the registration database building module 101 is used for registering all models needing centralized management in an enterprise to the same model management platform.

In an alternative embodiment, the registration library building module 101 may include the following elements:

a model definition creating unit 1011 for creating a model definition.

A metadata definition unit 1012 for defining model metadata.

And a data connection unit 1013 configured to connect an external database corresponding to the model with a data form of the model.

And the report processing unit 1014 is used for generating a model verification report according to the variables used by the data form selection model and giving report roles to the variables.

The model configuration unit 1015 is configured to configure a model, upload a model file corresponding to the model, define a data window and a reference, define a model prediction target, assign a report role-target to which the target belongs, and define a score distribution and a cutting group to which the model belongs.

An asset repository formation unit 1016 for saving all registered models to the enterprise level model asset repository.

And the asset sharing and multiplexing module 102 is configured to determine a matched available model in the model management platform according to the permissions corresponding to different platform roles, so that the platform roles share and multiplex the model assets based on the available model.

And the instrument panel monitoring module 103 is used for monitoring the working state, the performance index and the health state of the selected model in the life cycle of the model instrument panel.

In an alternative embodiment, the model dashboard displays the current status of all registered models and displays the verification results for the selected model in a preset display format including one or more of a sector statistical chart, bar statistical chart or table.

And the instrument board adjusting module 104 is used for adjusting the model state displayed on the model instrument board or the expression mode of the verification result according to the user requirement.

In an alternative embodiment, the workflow creation module 105 is used to create a generic model workflow based on any model work requirement or model work task.

And the workflow processing module 106 is used for performing model task processing on the general model workflow according to different model task requirements, wherein the model task at least comprises one or more of model requirement initiation, model development, model verification, model implementation and deployment, model approval, model online, continuous verification after production and monitoring subtasks.

In alternative implementations, the workflow creation module 106 may include the following elements:

a subtask creation unit 1061 for defining subtasks in the workflow.

And the workflow deployment unit 1062 is configured to deploy the issued workflow after the audit is correct, so that the workflow is automatically advanced according to the predefined task links and the responsible person.

And the subtask processing unit 1063 is configured to send, by the workflow, a reminding message to a person in charge corresponding to the next subtask according to the current progress, and assign a system right corresponding to the person in charge according to the set work in the system, where the reminding message may be any one or two of a mail reminding and a system login link, or may be other reminding manners.

And the subtask advancing unit 1064 is used for automatically advancing the workflow to the next subtask after one subtask is completed.

And a review feedback module 107 for feeding back historical activities and related records of the workflow during validation of the workflow according to review requests that comply with the workflow review authority.

And the updating and restarting module 108 is used for reissuing the workflow and restarting the task from the latest incomplete subtask when the workflow needs to be modified or the model task is added, deleted or updated during the validation of the workflow.

It should be noted that, for the detailed execution process of each module and unit in the system, reference may be made to the description in the foregoing embodiments, and details are not described here.

In the embodiment of the invention, by positioning at an enterprise level, an organization is assisted to establish an enterprise-level model asset library, models applied to different business scenes are managed in a centralized manner, and all models and related documents thereof are all registered on the same platform; the running state of the model or the working stage of the model can be known in real time according to the configured model instrument panel, a mechanism can know the working process of the model in time and follow up decision links such as artificial examination and approval required in a workflow, for the produced model, when the model is damaged or abnormal, an alarm rule is triggered and fed back to the model instrument panel in real time, and a user can intervene early to carry out model investigation; according to the given platform authority, a user can easily find needed model assets on the platform, the model assets are shared and reused in the mechanism, and the intelligent development is promoted by mutually referencing; based on the life cycle of the model, a model workflow of a universal version is defined, all model works can follow relatively consistent flows, each link of the workflow is assigned to a specific responsible person, work results needing to be uploaded are well defined according to different links, and examination and approval links are configured in the workflow according to needs, so that the work quality of each work link is guaranteed in a single model workflow, and the work quality requirements of the models are unified between the workflow and the workflow.

An embodiment of the present invention further provides a computer storage medium, where multiple instructions may be stored in the computer storage medium, where the instructions are suitable for being loaded by a processor and being executed in the method steps in the embodiments shown in fig. 1 to fig. 8, and specific execution processes may refer to specific descriptions in the embodiments shown in fig. 1 to fig. 8, which are not described herein again.

Fig. 12 is a schematic structural diagram of an apparatus according to an embodiment of the present invention. As shown in fig. 12, the apparatus 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, a user interface 1003, memory 1005, at least one communication bus 1002. The communication bus 1002 is used to implement connection communication among these components. The user interface 1003 may include a Display (Display) and a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface and a standard wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 1005 may optionally be at least one memory device located remotely from the processor 1001. As shown in fig. 12, memory 1005, which is one type of computer storage medium, may include an operating system, a network communication module, a user interface module, and an enterprise-level model management application.

In the apparatus 1000 shown in fig. 12, the user interface 1003 is mainly used as an interface for providing input for a user, and acquiring data input by the user; the network interface 1004 is used for data communication with the user terminal; and processor 1001 may be configured to invoke the enterprise-level model management application stored in memory 1005 and perform in particular the following:

and determining a matched available model in the model management platform according to the authorities corresponding to different platform roles so as to enable the platform roles to share and reuse model assets based on the available model.

In some embodiments, the processor 1001 performs the following operations when registering all models that need centralized management in an enterprise to the same model management platform:

creating a model definition;

defining model metadata;

connecting an external database corresponding to the model with a data form of the model;

generating a model verification report according to variables used by the data form selection model, and giving report roles to the variables;

configuring a model, uploading a model file corresponding to the model, defining a data window and a benchmark, defining a model prediction target, giving a report role-target to which the target belongs, and defining a score distribution and a cutting group to which the model belongs;

all registered models are saved to the enterprise level model asset library.

In some embodiments, the processor 1001 is further configured to perform the following operations:

monitoring the model operating state, performance index and health state of the selected model in the life cycle of the selected model in the model instrument panel;

in some embodiments, the model dashboard displays the current status of all registered models and displays the verification results for the selected model in a preset display form, including one or more of a sector statistical chart, a bar statistical chart, or a table.

In some embodiments, the processor 1001 is further configured to:

and adjusting the model state displayed on the model instrument board or the expression mode of the verification result according to the user requirement.

In some embodiments, the processor 1001 is further configured to:

creating a universal model workflow based on any model work requirement or model work task;

and performing model task processing on the general model workflow according to different model task requirements, wherein the model task at least comprises one or more of model requirement initiation, model development, model verification, model implementation and deployment, model approval, model online, continuous verification after production and monitoring subtasks.

In some embodiments, the processor 1001 specifically performs the following operations when creating a generic model workflow based on any model work requirement or model work task:

defining subtasks in the workflow;

after the audit is correct, the issued workflow is deployed so that the workflow can be automatically propelled according to the predefined task links and the responsible persons;

the workflow sends reminding information to a responsible person corresponding to the next subtask according to the current progress, and acquires system authority corresponding to the responsible person, wherein the reminding information comprises mail reminding and system login links;

after completion of one subtask, the workflow automatically advances to the next subtask.

during the workflow validation, feeding back the historical activities and related records of the workflow according to the query request conforming to the query authority of the workflow;

and when the workflow needs to be modified or the model tasks are added, deleted and updated during the validation of the workflow, the workflow is reissued and the tasks are restarted from the latest incomplete subtasks.

The model centralized management method is realized based on a model management system, and the model management system comprises the following equipment: the model monitoring system comprises a model unified management device, a model development device, a model operation device and a model monitoring device, wherein the model monitoring device and the model unified management device can be integrated into a whole. The above devices can be understood as a single computer device or a computer cluster, and are set according to specific application requirements.

The model development equipment is used for carrying out model development. The model operation device can be understood as a device which puts the model developed by the model development device into a business environment to operate and complete business requirements. The model development equipment is in communication connection with the model unified management equipment, the model unified management equipment configures a workflow for the model development equipment according to the development requirements of the model development equipment, and the model development equipment uploads the completion results of each task to the model unified management platform based on the workflow. Therefore, the model data in the workflow formation can be directly uploaded to the model unified management device, on one hand, the standardization of the model development process can be ensured, and on the other hand, the model uploaded to the model unified management device can be allowed to be shared and multiplexed.

The model monitoring equipment is in communication connection with the model operation equipment and the model unified management equipment, the monitoring needs to be configured through the model unified management equipment, and model parameters including but not limited to model production data are obtained from the model operation environment equipment according to the configured monitoring needs to monitor the model state. In order to display the monitoring result conveniently, the model monitoring equipment displays the parameters of each model through an instrument panel. Typically because each model has a data protocol and a debit of the respective model. In the centralized management method of the model, a unified management system is constructed, the development process of the model is started until the model is cancelled and fails, so that the model is unified, and on the other hand, the model monitoring equipment is configured with middleware for unifying protocols, so that the format of data is adjusted to be matched with the model monitoring equipment.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

The above disclosure is only for the purpose of illustrating the preferred embodiments of the present invention, and it is therefore to be understood that the invention is not limited by the scope of the appended claims.

Claims

1. A method for centralized management of models, the method comprising:

determining a matched available model in the model management platform according to the authority corresponding to different platform roles so that the platform roles can share and reuse model assets based on the available model;

model task processing is carried out on the general model workflow according to different model task requirements, wherein the model task at least comprises one or more of model requirement initiation, model development, model implementation and deployment, model verification, model approval, model online, continuous verification after production and monitoring subtasks;

wherein, registering all models needing centralized management in an enterprise to the same model management platform comprises: creating a model definition; defining model metadata; connecting an external database corresponding to the model with a data form of the model; generating a model verification report according to variables used by the data form selection model, and giving report roles to the variables; configuring a model, uploading a model file corresponding to the model, defining a data window and a benchmark, defining a model prediction target, giving a report role-target to which the target belongs, and defining a score distribution and a cutting group to which the model belongs; storing all registered models into an enterprise-level model asset library;

wherein, the creation of a generic model workflow based on any model work requirement or model work task comprises: creating a workflow and defining subtasks in the workflow; when the workflow is created, defining the name of the workflow, the type of the workflow and a corresponding general workflow template; when defining subtasks in the workflow, assigning the subtasks when creating the workflow according to the subtasks required by the model task; issuing a workflow, deploying the workflow, and advancing the workflow according to a defined task link and a responsible person; the workflow sends an email to a responsible person of the next subtask according to the progress, and a system work authority corresponding to the responsible person is given; when a responsible person in charge of the subtask finishes and uploads a document required by the task, the subtask is finished, the workflow automatically advances to the next subtask and informs related responsible persons, the workflow is repeated until all subtasks of the workflow are finished, whether the workflow is closed or not is determined according to the work quality and the workflow progress, and a finish label is printed; wherein a user having access to review the model or the workflow can review all historical activities and associated records during the workflow's validation; if the workflow is modified or updated during the workflow validation period, the workflow is reissued and redeployed according to the update, and the workflow restarts from the latest unfinished subtask and advances the work backwards;

the platform role at least comprises: the model belongs to people, workflow authors, model approval personnel and system administrators; different platform roles correspond to different permissions for completing different tasks;

the model owner is used for entering a platform system to inquire the states of all models managed by the model owner at present and entering a working environment of the model to inquire specific historical activities of a certain model or the present model expression condition;

the workflow author is used for entering a workflow management page of the platform system to create a model workflow according to a new model task, selecting a general workflow corresponding to the new model development task, wherein the general workflow covers subtask links of model development, model test, model approval, model deployment, model online and model verification, assigning a person to each subtask in the system, defining a required deliverable in each subtask, uploading the workflow after creation and waiting for system management deployment;

the model author is used for receiving a task mail and system connection from the system as the workflow advances after the workflow is officially deployed and worked; for a subtask-model development link, a model developer has the authority corresponding to the role of a model author in a system platform, completes data cleaning to generate a data quality report according to the requirements of the subtask, performs index derivation and feature screening to generate a feature report, performs model development and testing to generate a report, uploads the report to the subtask uniformly and submits the subtask;

the model approver is used for testing and verifying the model before the new model is online, checking the normative and the rationality of the reports in the stages of model development, model development and model verification, and giving approval for whether approval is passed or not, and when the model approver approves, the model can be continuously moved to the next link;

the system administrator is used for deploying the workflow after the workflow is created and uploaded by a workflow writer, taking effect after the workflow is deployed and displaying the workflow on a dashboard; aiming at a responsible person assigned by a sub-task in the workflow, a system administrator endows the responsible person assigned by the sub-task in the workflow with a corresponding system role and related authority;

the model management platform is associated with a workflow management system and a model verification/evaluation system, and the working relation between the workflow management system and the model verification/evaluation system comprises the following steps:

relation line a: model component-model author-model working environment-content management system; for an existing model, a model author creates a model definition in a model working environment, uploads components and related documents related to the model, and the components and the related documents of the model are stored in a content management system of a model risk management platform;

relation line b: model author-custom report-system administrator-system management-dashboard; the template of the self-defined report is designed outside a model risk management platform by a model author, then is uniformly managed and configured in the platform by a system administrator, when the configuration is completed, the model author utilizes the report to carry out verification and monitoring work in a model working environment, and the verification and monitoring result can be fed back to an instrument panel;

relation line c: the workflow author-workflow management-instrument board, the workflow author creates and maintains the workflow in the workflow management, when the workflow is released and deployed, the state of the workflow can be collected into the instrument board, and the workflow can supervise the model author to enter the model working environment to complete the task according to the work task and the responsible person in the workflow;

relation line d: system administrator-system management/rights management; the system administrator is responsible for managing a daily system platform, maintaining a platform environment, assisting a workflow writer to carry out workflow deployment, maintaining daily platform logs and managing roles of the platform and authorities corresponding to the roles;

relation line e: model production environment-model reflow data-model management platform database-model validation/monitoring work-validation report and results-content management system-model working environment; the model is called in a production environment to generate model backflow data, the backflow data is processed by ETL and flows back to a database externally connected with a model risk management platform, when a model author carries out model verification and monitoring work in the platform, a verification and monitoring engine in the platform calls data corresponding to the model in the externally connected database, and a verification report and a result are generated by combining model definition, variables and verification report role information in the platform and are stored in a content management system;

relation line f: model work environment-content management system-workflow management system; when the model author completes corresponding work or tasks in the model working environment, the completed work or operation is stored in the content management system, the content management system and the workflow management system are communicated with each other, the workflow system identifies the next action, pushes the workflow progress forward and returns the workflow progress to the content management system.

2. The method of model centralized management according to claim 1, wherein the method further comprises:

the model operating state, performance index and health of the selected model over its lifecycle are monitored within the model dashboard.

3. The model centralized management method according to claim 2, wherein the model dashboard displays the current status of all registered models and displays the verification results for the selected model in a preset display form, the display form including one or more of a sector statistical chart, a bar statistical chart or a table.

4. The method of model centralized management according to claim 2, wherein the method further comprises:

and adjusting the expression mode of the model state or the verification result displayed on the model instrument panel according to the user requirement.

5. The method of model centralized management according to claim 1, wherein the method further comprises:

a generic version of the model workflow is defined based on the model lifecycle.

6. The method for centralized model management according to claim 5, wherein the defining a generic version of the model workflow based on the model lifecycle comprises:

defining subtasks in the workflow;

after the examination is correct, the issued workflow is deployed so that the workflow is automatically propelled according to a predefined task link and a responsible person;

the workflow sends reminding information to a responsible person corresponding to the next subtask according to the current progress, and system permission required by a task link corresponding to the responsible person is given to the responsible person according to set work in the system, wherein the reminding information at least comprises one or more of mail reminding and system login links;

after completion of one subtask, the workflow advances automatically to the next subtask.

7. The method of model centralized management according to claim 6, wherein the method further comprises:

during the workflow validation, feeding back the historical activities and related records of the workflow according to the consulting request conforming to the consulting authority of the workflow;

8. A model centralized management system, the system comprising:

the instrument panel monitoring module is used for monitoring the working state, the performance index and the health state of the selected model in the life cycle of the model instrument panel;

the asset sharing and multiplexing module is used for determining a matched available model in the model management platform according to the authority corresponding to different platform roles so that the platform roles can share and multiplex model assets based on the available model;

the workflow creating module is used for creating a universal model workflow based on any model work requirement or model work task;

the workflow processing module is used for performing model task processing on the general model workflow according to different model task requirements, wherein the model task at least comprises one or more of model requirement initiation, model development, model verification, model implementation and deployment, model approval, model online, continuous verification after production and monitoring subtasks;

wherein, the registration library building module comprises: a model definition creating unit for creating a model definition; a metadata definition unit for defining model metadata; the data connection unit is used for connecting an external database corresponding to the model and a data form of the model; the report processing unit is used for generating a model verification report according to the variables used by the data form selection model and endowing the variables with report roles; the model configuration unit is used for configuring a model, uploading a model file corresponding to the model, defining a data window and a benchmark, defining a model prediction target, endowing a report role-target to which the target belongs, and defining a fraction distribution and a cutting group to which the model belongs; the asset library forming unit is used for storing all registered models into an enterprise-level model asset library;

the workflow creation module is specifically configured to: creating a workflow and defining subtasks in the workflow; when the workflow is created, defining the name of the workflow, the type of the workflow and a corresponding general workflow template; when the subtasks in the workflow are defined, assigning the subtasks when the workflow is established according to the subtasks required by the model task; issuing a workflow, deploying the workflow, and advancing the workflow according to a defined task link and a responsible person; the workflow sends an email to a responsible person of the next subtask according to the progress, and a system work authority corresponding to the responsible person is given; when a person in charge of the subtask completes and uploads a document required by the task, the subtask is completed, the workflow automatically advances to the next subtask and informs the related person in charge, the workflow is repeated until all subtasks of the workflow are completed, whether the workflow is closed or not is determined according to the work quality and the workflow progress, and a completion label is printed; wherein a user having access to review the model or the workflow can review all historical activities and associated records during the workflow's validation; if the workflow is modified or updated during the working period of the workflow, the workflow is reissued and redeployed according to the update, and the workflow is restarted from the latest unfinished subtask and pushed to work backwards;

the platform role at least comprises: the system comprises a person to whom the model belongs, a workflow writer, a model approval person and a system administrator; different platform roles correspond to different permissions for completing different tasks;

the system administrator is used for deploying the workflow after the workflow is created and uploaded by a workflow writer, taking effect after the workflow is deployed and displaying the workflow on a dashboard; aiming at a responsible person assigned by a subtask in the workflow, a system administrator gives a system role and related authority corresponding to the responsible person assigned by the subtask in the workflow;

relation line a: model component-model author-model work environment-content management system; for an existing model, a model author creates a model definition in a model working environment, uploads parts and related documents related to the model, and the parts and the related documents of the model are stored in a content management system of a model risk management platform;

a relation line c: the workflow author-workflow management-instrument board is used for creating and maintaining a workflow in the workflow management, when the workflow is released and deployed, the state of the workflow is collected into the instrument board, and the workflow supervises and urges the model author to enter a model working environment to complete a task according to a working task and a responsible person in the workflow;

relation line e: model production environment-model reflow data-model management platform database-model validation/monitoring work-validation report and results-content management system-model working environment; the model is called in a production environment to generate model backflow data, the backflow data is processed by an ETL (extract transform load) and flows back to a database externally connected with a model risk management platform, when a model author carries out model verification and monitoring work in the platform, a verification and monitoring engine in the platform calls data corresponding to the model in the externally connected database, and generates a verification report and a result by combining model definition, variables and verification report role information in the platform and stores the verification report and the result in a content management system;

relation line f: model work environment-content management system-workflow management system; when the model author completes corresponding work or task in the model working environment, the completed work or operation is stored in the content management system, the content management system and the workflow management system are communicated with each other, the workflow system identifies the next action, advances the workflow progress forward and returns the workflow progress to the content management system.