WO2023202067A1 - Object evaluation method and apparatus, storage medium, and electronic device - Google Patents

Abstract

The present disclosure relates to the field of computer application, and in particular to an object evaluation method and apparatus, a storage medium, and an electronic device. The object evaluation method comprises: in response to an evaluation instruction for a target evaluation object, obtaining an initial evaluation model and feature information of the target evaluation object; determining an evaluation index of a target level in the initial evaluation model on the basis of the feature information to obtain a final evaluation model, and determining weight information of the evaluation index of each level in the final evaluation model; obtaining an index score corresponding to the evaluation index of the target evaluation object at the target level; and according to the index score and the weight information, using the final evaluation model to perform layer-by-layer convergence to determine an evaluation result of the target evaluation object.

Description

Object evaluation methods, devices, storage media and electronic equipment

Cross-references to related applications

This application is based on the application with CN application number 202210411597.3 and the filing date is April 19, 2022, and claims its priority. The disclosure content of the CN application is hereby incorporated into this application as a whole.

Technical field

The present disclosure relates to the field of computer applications, and in particular, to an object evaluation method, an object evaluation device, a storage medium and an electronic device.

Background technique

With the continuous development of networks and the arrival of the 5G (5th Generation Mobile Communication Technology) era, operators' networks are becoming more and more complex, and business needs are gradually diversified, resulting in increasingly complex telecommunications 5G network operation and management. Aggravated. The introduction of artificial intelligence (AI) technology to enhance the intelligence and automation of telecommunications 5G network operation and management systems will have a significant impact on the existing operation and management system framework.

Assessing the level of the current telecom network management system helps operators evaluate their own intelligence level and provides guidance for network operation and maintenance evolution planning. Theories for assessing the level of autonomous or intelligent networks are currently being studied in ITU-T (ITU Telecommunication Standardization Sector) and other standards organizations.

It should be noted that the information disclosed in the above background section is only used to enhance understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to those of ordinary skill in the art.

Contents of the invention

The purpose of this disclosure is to provide an object evaluation method, object evaluation device, storage medium and electronic equipment, aiming to intelligently evaluate the current telecommunications network management system to provide guidance for network operation and maintenance evolution planning.

Additional features and advantages of the disclosure will be apparent from the following detailed description, or, in part, may be learned by practice of the disclosure.

According to an aspect of an embodiment of the present disclosure, an object evaluation method is provided, including: in response to an evaluation instruction of a target evaluation object, acquiring an initial evaluation model and characteristic information of the target evaluation object; determining the said target evaluation object based on the characteristic information. Initial evaluation of the evaluation indicators at the target level in the model to obtain the final evaluation model, and determining the weight information of the evaluation indicators at each level in the final evaluation model; obtaining the indicator scores corresponding to the evaluation indicators of the target evaluation object at the target level; The final evaluation model is used to perform layer-by-layer aggregation according to the indicator score and the weight information to determine the evaluation result of the target evaluation object.

According to some embodiments of the present disclosure, based on the foregoing solution, the initial evaluation model includes one or more of the secondary evaluation indicators among intention indicators, acquisition indicators, analysis indicators, decision indicators and execution indicators.

According to some embodiments of the present disclosure, based on the foregoing solution, the second-level evaluation index includes at least one third-level evaluation index; wherein the intention index includes an intention acquisition index, an intention identification index, an intention analysis index, an intention classification index, an intention One or more of the back-translation indicators; the collection indicators include data collection indicators and/or data processing indicators; the analysis indicators include one or more of resource analysis indicators, business analysis indicators, and customer analysis indicators; The decision-making indicators include policy generation indicators and/or execution judgment indicators; the execution indicators include system execution indicators and/or network element execution indicators.

According to some embodiments of the present disclosure, based on the foregoing solution, the evaluation indicators at the target level are four-level evaluation indicators; wherein the three-level evaluation indicators include at least one of the four-level evaluation indicators.

According to some embodiments of the present disclosure, based on the foregoing solution, the characteristic information includes: product business information, operation scenario information, and system scope information; wherein the product business information includes government and enterprise oriented categories, family oriented categories, and public individual oriented categories. One or more of categories and internal customer facing categories; the operation scenario information includes one or more of planning and construction, operation preparation and support, implementation delivery, guarantee and billing; the system scope information includes functions One or more of modules, feature sets, and entire systems.

According to some embodiments of the present disclosure, based on the foregoing solution, when the characteristic information is for IP virtual private networks in government and enterprise categories, service provisioning during implementation and delivery, and the entire system, the three-level evaluation indicators include at least one of the The four-level evaluation indicators include: the intention acquisition indicators include order acquisition indicators and/or product arrangement package acquisition indicators; the intention identification indicators include order identification indicators; the intention analysis indicators include order decomposition indicators; and the intention classification The indicators include capability mapping indicators; the intention back-translation indicators include completion feedback indicators; the data collection indicators include network element data collection indicators, bandwidth data collection indicators, business performance data collection indicators, pipeline resource data collection indicators, and process data collection One or more indicators and customer satisfaction collection indicators; the data processing indicators include data cleaning indicators and/or data verification indicators; the resource analysis indicators include idle port analysis indicators and/or pipeline resource analysis indicators; so The business analysis indicators include one or more of path calculation indicators, path analysis indicators, and completion test analysis indicators; the customer analysis indicators include activation experience analysis indicators; the strategy generation indicators include optimal path recommendation indicators, completion test analysis indicators, One or more of test plan generation indicators and strategy simulation verification indicators; the execution judgment indicators include path configuration decision indicators and/or completion test decision indicators; the system execution indicators include configuration command issuance indicators and/or test Command issuing indicators; the network element execution indicators include equipment network management execution indicators and/or equipment execution indicators.

According to some embodiments of the present disclosure, based on the foregoing solution, using the final evaluation model to perform layer-by-layer aggregation according to the indicator score and the weight information to determine the evaluation result of the target evaluation object includes: according to the The indicator score determines the fourth-level evaluation results corresponding to each of the four-level evaluation indicators; a weighted sum is performed based on the fourth-level evaluation results and the weight information of each of the four-level evaluation indicators to obtain the third-level evaluation corresponding to each third-level evaluation indicator. Result; Based on the weighted sum of the third-level evaluation results and the weight information of each third-level evaluation index, the second-level evaluation results corresponding to each second-level evaluation index are obtained; based on the second-level evaluation results and each second-level evaluation index The weight information of the evaluation indicators is weighted and summed to obtain the evaluation result of the target evaluation object.

According to a second aspect of the embodiment of the present disclosure, an object evaluation device is provided, including: a response module, configured to obtain an initial evaluation model and characteristic information of the target evaluation object in response to an evaluation instruction of the target evaluation object; a determination module , used to determine the evaluation indicators of the target level in the initial evaluation model based on the characteristic information to obtain the final evaluation model, and to determine the weight information of the evaluation indicators at each level in the final evaluation model; a scoring module, used to obtain the The index score corresponding to the evaluation index of the target evaluation object at the target level; an evaluation module, used to use the evaluation model to perform layer-by-layer aggregation according to the index score and the weight information to determine the evaluation of the target evaluation object result.

According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the object evaluation method in the above embodiment is implemented.

According to a fourth aspect of the embodiment of the present disclosure, an electronic device is provided, which is characterized in that it includes: one or more processors; a storage device for storing one or more programs. When the one or more programs When executed by the one or more processors, the one or more processors are caused to implement the object evaluation method as in the above embodiment.

According to a fifth aspect of an embodiment of the present disclosure, a computer program is provided, including: instructions that, when executed by a processor, cause the processor to perform the object evaluation method in the above embodiment.

It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and do not limit the present disclosure.

Description of the drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts. In the attached picture:

Figure 1 schematically shows a flow chart of an object evaluation method in an exemplary embodiment of the present disclosure;

Figure 2 schematically shows a composition diagram of an initial evaluation model in an exemplary embodiment of the present disclosure;

Figure 3 schematically shows a three-dimensional coordinate diagram of a type of feature information in an exemplary embodiment of the present disclosure;

Figure 4 schematically illustrates a flow chart of a method for determining an evaluation result in an exemplary embodiment of the present disclosure;

Figure 5 schematically shows the composition of an object evaluation device in an exemplary embodiment of the present disclosure;

Figure 6 schematically shows a schematic diagram of a computer-readable storage medium in an exemplary embodiment of the present disclosure;

FIG. 7 schematically shows a structural diagram of a computer system of an electronic device in an exemplary embodiment of the present disclosure.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concepts of the example embodiments. To those skilled in the art.

Furthermore, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to provide a thorough understanding of embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other instances, well-known methods, apparatus, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices. entity.

The flowcharts shown in the drawings are only illustrative, and do not necessarily include all contents and operations/steps, nor must they be performed in the order described. For example, some operations/steps can be decomposed, and some operations/steps can be merged or partially merged, so the actual order of execution may change according to the actual situation.

With the continuous development of networks and the arrival of the 5G era, operators' networks have become more and more complex, and business needs have gradually diversified, resulting in increasing complexity in telecommunications 5G network operation and management. The introduction of artificial intelligence (AI) technology to enhance the intelligence and automation of telecommunications 5G network operation and management systems will have a significant impact on the existing operation and management system framework. It is necessary to consider how to meet the requirements of network intelligence and achieve more efficient, economical, and flexible network operation and maintenance and service management. Assessing the level of the current telecom network management system helps operators evaluate their own intelligence level and provides guidance for network operation and maintenance evolution planning.

The theory of evaluating the level of autonomy or intelligent networks is currently being studied in ITU-T and other standards organizations, but there is currently no complete set of research on evaluating the intelligence level of telecommunications operation management systems. And the current research on network intelligence levels only provides guidance on Level 0-Level 5 and its basic characteristics. It does not provide evaluation measurement dimensions or parameters, making it difficult to be used in the assessment of the intelligence level of telecommunications network operation management systems.

ITU-T released a standard recommendation in 2020 on a framework for evaluating the intelligence level of future networks including IMT (IEMeeting, Emmett (Conference Equipment)-2020). It applies to the entire network and its operation and maintenance systems. However, , it only provides the theory of the method and does not propose how to evaluate the content of the telecom operation management system level. In the TMF (TeleManagement Forum, Telecommunications Management Forum) forum in February 2021, a standard draft on the autonomous network level was proposed. The The scope of the standard is to define the autonomous network level based on TMF Forum IG1218 (Autonomous Network Business Requirements and Architecture). It describes the concept of autonomous network level, including autonomous network level methodology and methods, operating procedures, underlying sub-processes and tasks, and task evaluation Standards. At present, the content of standard documents mainly focuses on the theory of intelligence level and does not involve operations and management.

In other words, there is currently no complete set of evaluation research on the intelligence level of telecommunications operation management systems. And the current research on network intelligence levels only provides guidance on Level 0-Level 5 and its basic characteristics. It does not provide evaluation measurement dimensions or parameters, making it difficult to use for intelligentization of telecommunications network operation and management systems. Level evaluation in progress.

Therefore, in view of the problems existing in the prior art, the present disclosure provides an object evaluation method, which can construct an analytic hierarchy process evaluation model based on the analytic hierarchy process model for the evaluation objects of the telecommunications operation management system in the IP virtual private network activation scenario, and Develop an indicator weight determination method to obtain the intelligent level classification (Level 0-Level 5) of the evaluation object at each level as the evaluation result.

The implementation details of the technical solutions of the embodiments of the present disclosure are described in detail below.

Figure 1 schematically shows a flow chart of an object evaluation method in an exemplary embodiment of the present disclosure. As shown in Figure 1, the object evaluation method includes steps S101 to S104:

Step S101, in response to the evaluation instruction of the target evaluation object, obtain the initial evaluation model and the characteristic information of the target evaluation object;

Step S102: Determine the evaluation indicators of the target level in the initial evaluation model based on the characteristic information to obtain the final evaluation model, and determine the weight information of the evaluation indicators at each level in the final evaluation model;

Step S103: Obtain the index score corresponding to the evaluation index of the target evaluation object at the target level;

Step S104: Use the evaluation model to perform layer-by-layer aggregation based on the indicator score and the weight information to determine the evaluation result of the target evaluation object.

In some embodiments, based on the indicator scores in the current level of the evaluation model, the indicator scores in the previous level are calculated; the previous level is determined as the current level, and the above steps are repeated until the indicators in the highest level of the evaluation model are calculated. Scoring; determine the evaluation result of the target evaluation object based on the indicator score in the highest level. For example, the target level can be determined as the initial current level, and the above-mentioned layer-by-layer aggregation process can be performed.

In the technical solution provided by some embodiments of the present disclosure, an initial evaluation model is first created, and after responding to the evaluation instruction, the characteristic information of the target evaluation object is automatically obtained; and then the target level in the evaluation model is determined based on the characteristic information of the target evaluation object. Corresponding evaluation indicators are used to obtain a complete final evaluation model, and the weight information of each evaluation indicator in the final evaluation model is determined; finally, the final evaluation model is used to gather the evaluation results layer by layer, so that a suitable and accurate evaluation can be carried out. Optimize the evaluation results to provide more accurate guidance for system optimization.

Below, each step of the object evaluation method in this exemplary embodiment will be described in more detail with reference to the accompanying drawings and embodiments.

In step S101, in response to an evaluation instruction of a target evaluation object, an initial evaluation model and characteristic information of the target evaluation object are obtained.

In some embodiments, after receiving the evaluation instruction, it is first necessary to obtain a pre-created initial evaluation model.

For example, an initial evaluation model that meets the evaluation needs of most evaluation objects can be created in advance. The initial evaluation model can include multiple levels of evaluation indicators, which are recorded as second-level, third-level and other evaluation indicators in turn. Each second-level evaluation indicator includes At least one third-level evaluation indicator, and so on for the others.

FIG. 2 schematically shows a composition diagram of an initial evaluation model in an exemplary embodiment of the present disclosure. As shown in Figure 2, the analytic hierarchy process evaluation model can be divided into four levels.

1) The first layer is used to store the characteristic information and evaluation results of the target evaluation object.

For example, the evaluation results can be obtained by aggregating and calculating the evaluation results corresponding to the secondary evaluation indicators, and the evaluation results are to be determined before the evaluation.

2) The second layer includes secondary evaluation indicators

For example, the second-level evaluation index is the evaluation index of the intent-driven workflow closed loop, including the five-dimensional closed-loop workflow from intention to execution, which can be obtained by aggregating and calculating the evaluation results corresponding to the third-level evaluation index. For example, the analytic hierarchy process evaluation model includes one or more of the secondary evaluation indicators among intention indicators, acquisition indicators, analysis indicators, decision-making indicators and execution indicators.

3) The third layer includes three-level evaluation indicators

For example, the third-level evaluation index is 14 evaluation sub-dimensions with general characteristics described based on the second-level evaluation index, which can be obtained by aggregating and calculating the evaluation results corresponding to the fourth-level evaluation index.

For example, the intention indicators include one or more of intention acquisition indicators, intention identification indicators, intention analysis indicators, intention classification indicators, and intention back-translation indicators; the collection indicators include data collection indicators and/or data processing indicators; The analysis indicators include one or more of resource analysis indicators, business analysis indicators, and customer analysis indicators; the decision-making indicators include strategy generation indicators and/or execution judgment indicators; the execution indicators include system execution indicators and/or Network element execution indicators.

4) The fourth level includes to-be-determined four-level evaluation indicators

For example, four-level evaluation indicators need to be defined based on the evaluation needs corresponding to different characteristic information of the evaluation object, and attached with rating rules. Different target evaluation objects have different characteristic information, so the four-level evaluation indicators here are to be determined and can only be confirmed after obtaining the characteristic information of the target evaluation object.

Therefore, in some embodiments, the target-level evaluation index is a fourth-level evaluation index; for example, the third-level evaluation index includes at least one of the fourth-level evaluation index. That is to say, after responding to the evaluation instruction of a certain target evaluation object, it is necessary to determine which fourth-level evaluation indicators are included in the third-level evaluation indicators based on the characteristic information of the target evaluation object.

In some embodiments, it is also necessary to obtain characteristic information of the target evaluation object. For example, the characteristic information includes: product business information, operation scenario information, and system scope information.

FIG. 3 schematically shows a three-dimensional coordinate diagram of a type of feature information in an exemplary embodiment of the present disclosure. As shown in Figure 3, the characteristic information of the target evaluation object mainly consists of three parts, namely product business, operation scenario, and system scope, which correspond to the x-axis, y-axis, and z-axis in Figure 3 respectively. Therefore, three-point focusing on the three axes of x, y, and z can uniquely determine an evaluation object, for example, marked with point coordinates, which can be recorded as (1, 1, 1).

For example, the product business information includes one or more of the categories for government and enterprises, for families, for the public and for individuals, and for internal customers. Below, it is introduced in detail:

For government and enterprise businesses: This type of service provides solutions built for enterprises. Drive your business forward with industry-leading business services. Work with the public sector to reimagine learning, modernize government, and transform networks. Typical products include VPN (Virtual Private Network) products, 5G slicing, IDC (Internet Data Center, Internet Data Center), etc.

Family-oriented business: This type of service is responsible for creating home entertainment, education and other services. Typical services include CDN (Content Delivery Network, content distribution network), broadband access, etc.

Businesses facing the public and individuals: This type of business needs to provide fast and reliable service solutions and is designed for small businesses, such as mobile positioning, 5G messaging, etc.

Businesses for internal customers: This type of business provides convenient daily operations and work solutions for internal customers, such as corporate offices, etc.

For example, the operation scenario information includes one or more of planning and construction, operation preparation and support, implementation delivery, guarantee and billing, which are described in detail below:

Plan and Build: Plan and Build is focused on the development and delivery of systems, computing resources, and network capabilities, including: defining, planning, and implementing all necessary infrastructure (such as applications, computing, and networks), as well as all other supporting infrastructure and business capabilities (such as partnerships, etc.). Plan and build products that identify needs, capabilities, and design and develop new or enhanced infrastructure to support operations

Operational preparation and support: Responsible for providing human resource allocation, management process and other management support for operations, and making operational preparations for the implementation of delivery, guarantee and billing benefits. For example, during product operation and maintenance, the call center directly faces customers, the configuration of back-end operation and maintenance personnel, front-line real-time response, and back-end operation processes, etc.

Implement Delivery: Responsible for delivering the products customers need in a timely and correct manner. It starts from the customer's purchase to the operator's delivery of products that meet the customer's needs, informs the customer of the status of the order, completes it on time, and ensures customer satisfaction.

Guarantee: Responsible for performing active and passive maintenance and optimization activities to ensure that continuously available products are provided to customers and to ensure SLA (Service-Level Agreement, service level agreement) or QoS (Quality of Service, providing service capabilities for designated network communications) performance . Ensure customer perception, service quality, and network performance, proactively detect and eliminate faults, predict potential problems, and optimize them. Report service quality to customers, receive reports from customers, notify customers of fault status, and ensure recovery and repair to ensure customer satisfaction.

Billing: Responsible for collecting customer product usage, determining charging and billing information, generating timely and accurate bills, providing customers with pre-billing usage information and billing, and processing their payments. Able to provide customer statements, provide bill inquiry status, and solve billing problems in a timely manner to satisfy customers. Prepayment is also supported.

It should be noted that the above five operational scenarios can be further refined into operational sub-scenarios based on actual operational conditions, such as including provisioning sub-scenarios in the implementation delivery scenario.

For example, the system-wide information includes one or more of functional modules, functional sets, and the entire system.

This refers to the dimensions considered during evaluation. The system includes multiple function sets, and each function set is composed of different functional modules. When conducting evaluation, you can evaluate only the dimensions of a single functional module of a certain product business in a certain operating scenario, or you can make a rating based on the entire system architecture, making the evaluation more flexible.

In step S102, the evaluation indicators of the target level in the initial evaluation model are determined based on the characteristic information to obtain the final evaluation model, and the weight information of the evaluation indicators of each level in the final evaluation model is determined.

In some embodiments, the evaluation indicators at the target level in the initial evaluation model are determined, that is, the four-level evaluation indicators included in each three-level evaluation indicator are determined.

For example, when the characteristic information of the target evaluation object is that the product business information is an IP virtual private network for government and enterprises, the operation scenario information is the service provisioning sub-scenario in implementation delivery, and the system scope information is the entire system, the other The corresponding final evaluation model is shown in Table 1.

Table 1 Final evaluation model

For example, the three-level evaluation indicators include the four-level evaluation indicators:

The intention acquisition indicators include order acquisition indicators and/or product orchestration package acquisition indicators; the intention identification indicators include order identification indicators; the intention analysis indicators include order decomposition indicators; the intention classification indicators include capability mapping indicators; Intention back-translation indicators include as-built feedback indicators;

The data collection indicators include one or more of network element data collection indicators, bandwidth data collection indicators, business performance data collection indicators, pipeline resource data collection indicators, process data collection indicators, and customer satisfaction collection indicators; Data processing indicators include data cleaning indicators and/or data verification indicators;

The resource analysis indicators include idle port analysis indicators and/or pipeline resource analysis indicators; the business analysis indicators include one or more of path calculation indicators, path analysis indicators, and completion test analysis indicators; the customer analysis indicators Including activation experience analysis indicators;

The strategy generation indicators include one or more of optimal path recommendation indicators, completion test plan generation indicators, and strategy simulation verification indicators; the execution judgment indicators include path configuration decision indicators and/or completion test decision indicators;

The system execution indicators include configuration command issuance indicators and/or test command issuance indicators; the network element execution indicators include equipment network management execution indicators and/or equipment execution indicators.

In addition to determining the evaluation indicators, the evaluation rules for the indicators must also be formulated so that the corresponding indicator scores for the indicators can be obtained later.

It should be noted that different four-level evaluation indicators and evaluation rules need to be formulated according to different characteristic information of the evaluation object, so they can be formulated in advance. After completion, the characteristic information and the corresponding four-level evaluation indicators and evaluation rules will be mapped and stored, and then After obtaining the feature information, the four-level evaluation indicators and evaluation rules corresponding to the feature information can be mapped.

In some embodiments, it is also necessary to determine the weight information of the evaluation indicators at each level in the final evaluation model.

For example, since the evaluation model is constructed with multiple levels of evaluation indicators, the analytic hierarchy process is used to determine the weight of each evaluation indicator. Determining the weights involves the following steps.

Step 1: Construct a judgment matrix. Assume that the index of a certain layer is F = [f ₁ , f ₂ ,..., f _n ], compare the importance of the influence of f _i and f _j on a certain index of the upper layer in pairs, expressed by f _ij = f _i /f _j , f The value of _ij is quantified with numbers 1 to 9 and their reciprocals, and finally the judgment matrix F(f _ij ) _n×n is formed.

Step 2: After constructing the hierarchically sorted single-layer judgment matrix, calculate the maximum eigenvalue λ _max of the judgment matrix F and its corresponding eigenvectors, and then normalize the eigenvectors to obtain the importance of each indicator in the same layer relative to an indicator in the upper layer. In order to obtain the ranking weight of each indicator at the lowest level for the overall goal, the total ranking of the hierarchy is calculated. The total ranking weight combines the single ranking weights of the hierarchy from top to bottom.

Step 3: Consistency check. The consistency ratio of the judgment matrix C _R is determined by the ratio of the consistency index C ₁ and the degree of freedom index R _1. When C _R <0.1, the consistency of the judgment matrix is within a reasonable range, otherwise it should be corrected, C ₁ = (λ _max ()(n-1)), C _R =C ₁ /R ₁ .

Step 4: Solve the judgment matrix weight. Use the eigenvector method to solve the weight vector for the judgment matrix, where FW = λ′ _max , and solve for W′ to get W′ _l = [w′ _l,1 ,w′ _l,2 ,…,w′ _l,n ] ^T , for W′ is normalized, and the relative weight of each indicator in a certain layer to the indicator in the upper layer can be obtained according to the following formula, as shown in formula (1):

In step S103, the index score corresponding to the evaluation index of the target evaluation object at the target level is obtained.

In some embodiments, the target-level evaluation indicators are four-level evaluation indicators. According to the established evaluation rules, the indicator scores corresponding to each four-level evaluation indicator are determined based on expert experience.

Of course, in addition to the expert scoring method, other evaluation methods can also be used, such as scoring or rating based on test questions. The embodiments of the present disclosure are only illustrative and do not limit the present disclosure.

In step S104, the evaluation model is used to perform layer-by-layer aggregation according to the indicator score and the weight information to determine the evaluation result of the target evaluation object.

FIG. 4 schematically illustrates a flow chart of a method for determining an evaluation result in an exemplary embodiment of the present disclosure. As shown in Figure 4, determining the evaluation results includes the following steps:

Step S401: Determine the four-level evaluation results corresponding to each of the four-level evaluation indicators based on the indicator scores;

Step S402: Perform a weighted sum based on the four-level evaluation results and the weight information of each of the four-level evaluation indicators to obtain the three-level evaluation results corresponding to each three-level evaluation indicator;

Step S403: Perform a weighted sum based on the third-level evaluation results and the weight information of each of the third-level evaluation indicators to obtain the second-level evaluation results corresponding to each second-level evaluation indicator;

Step S404: Perform a weighted sum based on the secondary assessment results and the weight information of each secondary assessment index to obtain an assessment result of the target assessment object.

For example, the evaluation results can use the intelligence level of Level 0-Level 5 to describe the evaluation results of the target evaluation object. In actual operation, the scores of the fourth-level evaluation indicators can first be mapped to the level rating results, and then the weighted summation is performed based on the weight information of the evaluation indicators, and the smart level grades of each third-level and second-level evaluation indicators are calculated in turn. , and finally the intelligence level of the target evaluation object is obtained. The calculation results are shown in Table 2:

Table 2 Intelligence level evaluation results

As shown in Table 2, first, the expert scores of the four-level evaluation indicators are mapped to the level rating results, and then the hierarchical model ratings are aggregated layer by layer based on the weight information in the final evaluation model to calculate the IP virtual private network service. In this scenario, the intelligence level evaluation and classification result of the entire system is 1.06.

In other embodiments, the characteristic information of the target evaluation object is different, and the evaluation results of the final evaluation model and the intelligence level are different. For example, Table 3 shows:

Table 3 Intelligence level evaluation results

As shown in Table 3, when the characteristic information is: product business information is IP virtual private network, operation scenario information is VPN service guarantee, and system scope information is the entire system, the intelligence level is 1.85.

Therefore, based on the above method, on the one hand, an intelligent level evaluation framework is constructed based on the analytic hierarchy process model, the evaluation indicators are defined and decomposed layer by layer, and the weights of indicators at each level are further determined to provide a complete set of system intelligent level evaluation methods and devices. It can promote the intelligent evolution of the entire operation management system and guide the implementation of the intelligent evolution of the system, which has strong guiding significance; on the other hand, based on the method of determining the three-dimensional coordinate evaluation object corresponding to the characteristic information, it can clearly define the telecom operation management system. Intelligence level evaluation is inseparable from the prerequisites of product business and operational scenarios.

FIG. 5 schematically shows the composition of an object evaluation device in an exemplary embodiment of the present disclosure. As shown in FIG. 5 , the object evaluation device 500 may include a response module 501 , a determination module 502 , a scoring module 503 and an evaluation module 504 . in:

The response module 501 is configured to respond to the evaluation instruction of the target evaluation object and obtain the initial evaluation model and the characteristic information of the target evaluation object;

Determining module 502, configured to determine the evaluation indicators of the target level in the initial evaluation model based on the characteristic information to obtain the final evaluation model, and determine the weight information of the evaluation indicators at each level in the final evaluation model;

The scoring module 503 is used to obtain the index score corresponding to the evaluation index of the target evaluation object at the target level;

The evaluation module 504 is configured to use the evaluation model to perform layer-by-layer aggregation according to the indicator score and the weight information to determine the evaluation result of the target evaluation object.

According to an exemplary embodiment of the present disclosure, the initial evaluation model includes one or more of the secondary evaluation indicators among intent indicators, collection indicators, analysis indicators, decision indicators and execution indicators.

According to an exemplary embodiment of the present disclosure, the second-level evaluation index includes at least one third-level evaluation index; for example, the intent index includes an intent acquisition index, an intent identification index, an intent parsing index, an intent classification index, and an intent back-translation index. One or more of the above; the collection indicators include data collection indicators and/or data processing indicators; the analysis indicators include one or more of resource analysis indicators, business analysis indicators, and customer analysis indicators; the decision-making The indicators include policy generation indicators and/or execution judgment indicators; the execution indicators include system execution indicators and/or network element execution indicators.

According to an exemplary embodiment of the present disclosure, the target-level evaluation index is a fourth-level evaluation index; for example, the third-level evaluation index includes at least one of the fourth-level evaluation index.

According to an exemplary embodiment of the present disclosure, the characteristic information includes: product business information, operation scenario information, and system scope information; for example, the product business information includes: oriented to government and enterprise categories, oriented to family categories, oriented to public individuals, and oriented to One or more of the internal customer categories; the operation scenario information includes one or more of planning and construction, operation preparation and support, implementation delivery, guarantee and billing; the system-wide information includes functional modules, functions One or more of a collection and an entire system.

According to an exemplary embodiment of the present disclosure, when the characteristic information is for IP virtual private networks in the government and enterprise category, service provisioning during implementation and delivery, and the entire system, the three-level evaluation indicators include at least one of the four-level Evaluation indicators include: the intention acquisition indicators include order acquisition indicators and/or product orchestration package acquisition indicators; the intention identification indicators include order identification indicators; the intention analysis indicators include order decomposition indicators; the intention classification indicators include capabilities Mapping indicators; the intention back-translation indicators include completion feedback indicators; the data collection indicators include network element data collection indicators, bandwidth data collection indicators, business performance data collection indicators, pipeline resource data collection indicators, process data collection indicators, customer One or more of the satisfaction collection indicators; the data processing indicators include data cleaning indicators and/or data verification indicators; the resource analysis indicators include idle port analysis indicators and/or pipeline resource analysis indicators; the business analysis The indicators include one or more of path calculation indicators, path analysis indicators, and completion test analysis indicators; the customer analysis indicators include activation experience analysis indicators; the strategy generation indicators include optimal path recommendation indicators, and completion test plan generation One or more of indicators and strategy simulation verification indicators; the execution judgment indicators include path configuration decision indicators and/or completion test decision indicators; the system execution indicators include configuration command issuance indicators and/or test command issuance indicators Indicators; the network element execution indicators include equipment network management execution indicators and/or equipment execution indicators.

According to an exemplary embodiment of the present disclosure, the evaluation module 504 is configured to determine a fourth-level evaluation result corresponding to each of the four-level evaluation indicators according to the indicator score; based on the fourth-level evaluation result and each of the four-level evaluation Perform a weighted summation of the weight information of the indicators to obtain the third-level evaluation results corresponding to each third-level evaluation indicator; perform a weighted sum based on the third-level evaluation results and the weight information of each of the third-level evaluation indicators to obtain the corresponding second-level evaluation indicators. The secondary evaluation results; based on the secondary evaluation results and the weight information of each of the secondary evaluation indicators, a weighted sum is performed to obtain the evaluation result of the target evaluation object.

The specific details of each module in the above-mentioned object evaluation device 500 have been described in detail in the corresponding xxx method, so they will not be described again here.

It should be noted that although several modules or units of equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided into being embodied by multiple modules or units.

In an exemplary embodiment of the present disclosure, a storage medium capable of implementing the above method is also provided. Figure 6 schematically shows a schematic diagram of a computer-readable storage medium in an exemplary embodiment of the present disclosure. As shown in Figure 6, a program product 600 for implementing the above method according to an embodiment of the present disclosure is described, which can It adopts a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on a terminal device such as a mobile phone. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. FIG. 7 schematically shows a structural diagram of a computer system of an electronic device in an exemplary embodiment of the present disclosure.

It should be noted that the computer system 700 of the electronic device shown in FIG. 7 is only an example, and should not impose any restrictions on the functions and scope of use of the embodiments of the present disclosure.

As shown in Figure 7, the computer system 700 includes a central processing unit (Central Processing Unit, CPU) 701, which can be loaded into a random accessory according to a program stored in a read-only memory (Read-Only Memory, ROM) 702 or from a storage part 708. Access the program in the memory (Random Access Memory, RAM) 703 to perform various appropriate actions and processes. In RAM 703, various programs and data required for system operation are also stored. CPU 701, ROM 702 and RAM 703 are connected to each other through bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

The following components are connected to the I/O interface 705: an input part 706 including a keyboard, a mouse, etc.; an output part 707 including a cathode ray tube (Cathode Ray Tube, CRT), a liquid crystal display (Liquid Crystal Display, LCD), etc., and a speaker, etc. ; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. Driver 710 is also connected to I/O interface 705 as needed. Removable media 711, such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

In particular, according to embodiments of the present disclosure, the processes described below with reference to the flowcharts may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such embodiments, the computer program may be downloaded and installed from the network via communication portion 709 and/or installed from removable media 711 . When this computer program is executed by the central processing unit (CPU) 701, various functions defined in the system of the present disclosure are performed.

It should be noted that the computer-readable medium shown in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read only memory (ROM), removable Programmable Read-Only Memory (Erasable Programmable Read Only Memory, EPROM), flash memory, optical fiber, portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage device, magnetic storage device, or any of the above suitable The combination. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device . Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved. It will also be noted that each block in the block diagram or flowchart illustration, and combinations of blocks in the block diagram or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or may be implemented by special purpose hardware-based systems that perform the specified functions or operations. Achieved by a combination of specialized hardware and computer instructions.

The units involved in the embodiments of the present disclosure can be implemented in software or hardware, and the described units can also be provided in a processor. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances.

As another aspect, the present disclosure also provides a computer-readable medium. The computer-readable medium may be included in the electronic device described in the above embodiments; it may also exist independently without being assembled into the electronic device. middle. The computer-readable medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiments.

It should be noted that although several modules or units of equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided into being embodied by multiple modules or units.

Through the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described here can be implemented by software, or can be implemented by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or on the network , including several instructions to cause a computing device (which may be a personal computer, a server, a touch terminal, a network device, etc.) to execute the method according to the embodiments of the present disclosure.

Other embodiments of the disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the disclosure that follow the general principles of the disclosure and include common common sense or customary technical means in the technical field that are not disclosed in the disclosure. .

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the disclosure is limited only by the appended claims.

Claims (15)

1. An object evaluation method that includes:

   In response to the evaluation instruction of the target evaluation object, obtain the initial evaluation model and the characteristic information of the target evaluation object;

   Based on the characteristic information, determine the evaluation indicators of the target level in the initial evaluation model to obtain the final evaluation model, and determine the weight information of the evaluation indicators at each level in the final evaluation model;

   Obtain the index score corresponding to the evaluation index of the target evaluation object at the target level;

   According to the indicator score and the weight information, the final evaluation model is used to perform layer-by-layer aggregation to determine the evaluation result of the target evaluation object.
2. The object evaluation method according to claim 1, wherein the initial evaluation model includes a secondary evaluation indicator, and the secondary evaluation indicator includes one of an intention indicator, a collection indicator, an analysis indicator, a decision indicator and an execution indicator, or Various.
3. The object evaluation method according to claim 2, wherein:

   The second-level evaluation index includes at least one third-level evaluation index;

   The three-level evaluation indicators included in the intention indicators include intention acquisition indicators and intention recognition indicators.
4. The object evaluation method according to claim 3, wherein the third-level evaluation index included in the intention index includes one or more of an intention analysis index, an intention classification index, or an intention back-translation index.
5. The object evaluation method according to claim 3, wherein:

   The three-level evaluation indicators included in the collection indicators include one or more of data collection indicators or data processing indicators;

   The third-level evaluation indicators included in the analysis indicators include one or more of resource analysis indicators, business analysis indicators or customer analysis indicators;

   The three-level evaluation indicators included in the decision-making indicators include one or more of strategy generation indicators or execution judgment indicators;

   The third-level evaluation indicators included in the execution indicators include one or more of system execution indicators or network element execution indicators.
6. The object evaluation method according to any one of claims 3 to 5, wherein the target level evaluation index is a fourth-level evaluation index, and the third-level evaluation index includes at least one of the fourth-level evaluation index.
7. The object evaluation method according to claim 1, wherein:

   The characteristic information includes product business information, operation scenario information, and system scope information;

   The product business information includes one or more of the categories for government and enterprises, for families, for the public and for individuals, or for internal customers;

   The operation scenario information includes one or more of planning and construction, operation preparation and support, implementation delivery or guarantee, and billing;

   The system-wide information includes one or more of functional modules, functional sets, or the entire system.
8. The object evaluation method according to claim 6, wherein when the characteristic information includes IP virtual private networks in the government and enterprise category, service provisioning in implementation and delivery, and the entire system, the intention acquisition indicator includes The fourth-level evaluation index includes at least one of an order acquisition index or a product orchestration package acquisition index, and the fourth-level evaluation index included in the intent identification index includes an order identification index.
9. The object evaluation method according to claim 6, wherein when the characteristic information includes IP virtual private networks in the government and enterprise category, service provisioning in implementation and delivery, and the entire system, the intent analysis index includes The fourth-level evaluation indicators include order decomposition indicators, the fourth-level evaluation indicators included in the intention classification indicators include capability mapping indicators, and the fourth-level evaluation indicators included in the intention back-translation indicators include completion feedback indicators.
10. The object evaluation method according to claim 6, wherein:

    The third-level evaluation indicators included in the data processing indicators include one or more of data cleaning indicators or data verification indicators;

    The three-level evaluation indicators included in the resource analysis indicators include one or more of idle port analysis indicators or pipeline resource analysis indicators;

    The third-level evaluation indicators included in the business analysis indicators include one or more of path calculation indicators, path analysis indicators, or completion test analysis indicators;

    The three-level evaluation indicators included in the customer analysis indicators include activation experience analysis indicators;

    The three-level evaluation indicators included in the strategy generation indicators include one or more of optimal path recommendation indicators, completion test plan generation indicators, or strategy simulation verification indicators;

    The third-level evaluation indicators included in the execution judgment indicators include one or more of path configuration decision indicators or completion test decision indicators;

    The three-level evaluation indicators included in the system execution indicators include one or more of configuration command issuance indicators or test command issuance indicators;

    The third-level evaluation indicators included in the network element execution indicators include one or more of equipment network management execution indicators or equipment execution indicators.
11. The object evaluation method according to claim 6, wherein, using the final evaluation model to perform layer-by-layer aggregation according to the indicator score and the weight information to determine the evaluation result of the target evaluation object includes:

    Determine the four-level evaluation results corresponding to each of the four-level evaluation indicators based on the indicator scores;

    Perform a weighted sum based on the four-level assessment results and the weight information of each of the four-level assessment indicators to obtain the three-level assessment results corresponding to each three-level assessment indicator;

    Perform a weighted sum based on the third-level evaluation results and the weight information of each of the third-level evaluation indicators to obtain the second-level evaluation results corresponding to each second-level evaluation indicator;

    The evaluation result of the target evaluation object is obtained by performing a weighted sum based on the secondary evaluation results and the weight information of each of the secondary evaluation indicators.
12. An object evaluation device comprising:

    A response module, configured to respond to the evaluation instruction of the target evaluation object and obtain the initial evaluation model and the characteristic information of the target evaluation object;

    A determination module, configured to determine the evaluation indicators of the target level in the initial evaluation model based on the characteristic information to obtain the final evaluation model, and determine the weight information of the evaluation indicators at each level in the final evaluation model;

    A scoring module, used to obtain the index score corresponding to the evaluation index of the target evaluation object at the target level;

    An evaluation module is configured to use the final evaluation model to perform layer-by-layer aggregation according to the indicator score and the weight information to determine the evaluation result of the target evaluation object.
13. A computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the object evaluation method according to any one of claims 1 to 11 is implemented.
14. An electronic device including:

    one or more processors;

    Storage device, used to store one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement any one of claims 1 to 11 The object evaluation method described in the item.
15. A computer program consisting of:

    Instructions which, when executed by a processor, cause the processor to perform the object evaluation method according to any one of claims 1 to 11.

Images