CN117714300A

CN117714300A - Evaluation method, device and system for autonomous level of operation and maintenance network

Info

Publication number: CN117714300A
Application number: CN202211098293.2A
Authority: CN
Inventors: 刘银萍; 李贤明
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2022-09-08
Filing date: 2022-09-08
Publication date: 2024-03-15
Also published as: WO2024051805A1

Abstract

The application provides an operation and maintenance network autonomous level assessment method, device and system, and relates to the technical field of communication. The first device acquires the evaluation data of N operation and maintenance tasks in the first operation and maintenance operation link from the second device, the first device further sends the evaluation data of N operation and maintenance tasks to the third device, and the third device can determine the network autonomy level of the first operation and maintenance operation link according to the evaluation data of N operation and maintenance tasks; or the first device determines the network autonomy level of the first operation and maintenance operation link according to the evaluation data of the N operation and maintenance tasks. In the method, the network autonomous level of the operation link is collected and evaluated based on the evaluation data of the operation task, so that the reliability of the evaluation result can be improved.

Description

Evaluation method, device and system for autonomous level of operation and maintenance network

Technical Field

The embodiment of the application relates to the technical field of communication, in particular to an evaluation method, an evaluation device and an evaluation system for an operation and maintenance network autonomous level.

Background

With the development of communication technology, the number of terminal devices in a communication network is rapidly increased, the service range is gradually expanded, the communication network is more and more complex, and the operation and maintenance difficulty of the network is increased. How to reduce the network operation and maintenance cost, simplify the network operation and maintenance flow, and rapidly deploy the network to meet diversified services is a problem to be solved in the network operation and maintenance field. Operators are willing to introduce autonomous techniques (e.g., artificial intelligence, machine learning, big data analysis) in various scenarios of network lifecycle, including network planning, network deployment, network optimization, traffic operation, etc., to reduce manual operations and improve operation and maintenance efficiency.

The autonomous level of the operation and maintenance network reflects the operation and maintenance efficiency and operation and maintenance effect and can be used as an important index for measuring the performance of network products of operators. At present, the operation and maintenance network provider provides the evaluation result of the autonomous level of the operation and maintenance network, and the mode is not transparent enough and has poor reliability.

Disclosure of Invention

The embodiment of the application provides an operation and maintenance network autonomous level evaluation method, device and system, so as to improve reliability of operation and maintenance network autonomous level evaluation.

In a first aspect, an embodiment of the present application provides a method for evaluating an autonomous level of an operation and maintenance network, including:

the first device acquires evaluation data of N operation and maintenance tasks in a first operation and maintenance operation link from the second device, wherein N is a positive integer. The first device sends evaluation data of the N operation and maintenance tasks to a third device, wherein the evaluation data of the N operation and maintenance tasks are used for determining the network autonomy level of the first operation and maintenance operation link; or the first device determines the network autonomy level of the first operation link according to the evaluation data of the N operation tasks.

In the design, the evaluation data of the interactive operation and maintenance task among the devices are supported, the network autonomy level of the operation and maintenance operation link is evaluated based on the evaluation data of the operation and maintenance task, an objective and transparent evaluation scheme can be realized, and the reliability of an evaluation result is improved.

In one possible design, the first device may further send first information to the second device, where the first information is used to request evaluation data of N operation tasks in the first operation link, and N is a positive integer. And the second device sends the evaluation data of N operation and maintenance tasks in the first operation and maintenance operation link to the first device according to the first information. In the design, the first device controls the second device to send the evaluation data of the operation and maintenance tasks, so that the first device can acquire the evaluation data of the required operation and maintenance tasks, and the fairness and the reliability of an evaluation scheme can be ensured.

In one possible design, the first device receives the second information from the third device. The second information is used for requesting to evaluate the network autonomy level of the first operation link, and the second information comprises the first information. The design supports the third device to trigger the evaluation of the operation and maintenance operation links, and can apply Yu Yunwei network providers to obtain the evaluation data or the scene of the evaluation result through the third device outside, and support objective evaluation through various evaluation approaches. Optionally, the second information includes content in the first information, but the signaling format of the first information is different from that of the second information; alternatively, the first information is encapsulated in the second information, and the first information is a part of the second information.

In one possible design, the evaluation data of any one of the N operation and maintenance tasks of the first information request includes one or more of the following: the information of any operation and maintenance task, the execution mode of any operation and maintenance task, and the execution effect of any operation and maintenance task. The design clearly shows the type of the evaluation data, is convenient for the second device to quickly and effectively collect the required evaluation data, and is beneficial to improving the evaluation efficiency.

The first device requests the evaluation data of different operation and maintenance tasks in the N operation and maintenance tasks through the first information, wherein the evaluation data of different operation and maintenance tasks in the N operation and maintenance tasks can be the same or different. For example, N is 5, and the first information request includes information of the operation and maintenance task and an execution mode for the evaluation data of each of the 3 operation and maintenance tasks; the first information requests evaluation data of each of the remaining 2 operation tasks including information of the operation task, an execution mode, and an execution effect.

In one possible design, the first information includes index information for measuring the execution effect of some or all of the N operation and maintenance tasks. The different operation and maintenance tasks correspond to index information of different measurement execution effects.

In one possible design, the first information includes first indication information, where the first indication information is used to indicate a candidate mode corresponding to an execution mode of any one of the N operation and maintenance tasks. Wherein the candidate patterns include: a manual execution mode, a manual and system common execution mode, and a system execution mode. The design definitely defines the execution mode, can simplify the indication of the execution mode and saves the signaling overhead.

In one possible design, the first device obtains, from the second device, evaluation data of N operation tasks in the first operation link, including: the first device receives third information from the second device, the third information indicating one or more of: and the processing of the N operation and maintenance tasks is completed, the execution mode of the N operation and maintenance tasks is completed, and the execution effect of part or all of the N operation and maintenance tasks is achieved.

In a second aspect, an embodiment of the present application provides a method for evaluating an autonomous level of an operation and maintenance network, including:

the second device receives first information from the first device, wherein the first information is used for requesting evaluation data of N operation and maintenance tasks in a first operation and maintenance operation link, and N is a positive integer. And the second device sends third information to the first device according to the first information, wherein the third information is used for indicating the evaluation data of the N operation and maintenance tasks.

In one possible design, the evaluation data of any one of the N operation and maintenance tasks of the first information request includes one or more of the following: the information of any operation and maintenance task, the execution mode of any operation and maintenance task, and the execution effect of any operation and maintenance task.

In one possible design, the first information includes first indication information, where the first indication information is used to indicate a candidate mode corresponding to an execution mode of any one of the N operation and maintenance tasks, and the candidate mode includes: a manual execution mode, a manual and system common execution mode, and a system execution mode.

In one possible design, the third information is used to indicate one or more of the following: and the processing of the N operation and maintenance tasks is completed, the execution mode of the N operation and maintenance tasks is completed, and the execution effect of part or all of the N operation and maintenance tasks is achieved.

In a third aspect, an embodiment of the present application provides a method for evaluating an autonomous level of an operation and maintenance network, including: the second device sends evaluation data of N operation and maintenance tasks in the first operation and maintenance operation link to the first device, wherein N is a positive integer. The first device determines the network autonomy level of the first operation link according to the evaluation data of the N operation tasks; or the first device sends the evaluation data of the N operation and maintenance tasks to a third device, and the third device determines the network autonomy level of the first operation and maintenance operation link according to the evaluation data of the N operation and maintenance tasks.

In one possible design, the method further comprises: the first device sends first information to the second device, wherein the first information is used for requesting evaluation data of N operation and maintenance tasks in a first operation and maintenance operation link.

In one possible design, the evaluation data of the N operation and maintenance tasks of the first information request includes one or more of the following: the information of the N operation and maintenance tasks, the execution mode of the N operation and maintenance tasks, and the execution effect of part or all operation and maintenance tasks in the N operation and maintenance tasks.

In one possible design, the first information includes first indication information, where the first indication information is used to indicate a candidate mode corresponding to an execution mode of any operation and maintenance task in the N operation and maintenance tasks. Wherein the candidate patterns include: a manual execution mode, a manual and system common execution mode, and a system execution mode.

In one possible design, the method further comprises: the third device sends second information to the first device. The second information is used for requesting to evaluate the network autonomy level of the first operation link, and the second information comprises the first information.

In one possible design, the second device sends, to the first device, evaluation data of N operation tasks in the first operation link, including: the second device sends third information to the first device, the third information being used to indicate one or more of: and the processing of the N operation and maintenance tasks is completed, the execution mode of the N operation and maintenance tasks is completed, and the execution effect of part or all of the N operation and maintenance tasks is achieved.

In a fourth aspect, an embodiment of the present application provides an evaluation device for an autonomous level of an operation and maintenance network, where the device may be a first device, or may be a module or a chip in the first device, or may be a device that can be used in a matching manner with the first device. In one design, the apparatus may include modules corresponding to the methods/operations/steps/actions described in the first aspect, where the modules may be hardware circuits, software, or a combination of hardware circuits and software. In one design, the apparatus may include a processing module and a communication module. Comprising the following steps:

the processing module is used for acquiring the evaluation data of N operation and maintenance tasks in the first operation and maintenance operation link from the second device, wherein N is a positive integer.

The communication module is used for sending the evaluation data of the N operation and maintenance tasks to a third device, wherein the evaluation data of the N operation and maintenance tasks are used for determining the network autonomy level of the first operation and maintenance operation link; or,

and the processing module is also used for determining the network autonomy level of the first operation link according to the evaluation data of the N operation tasks.

In one possible design, the communication module is further configured to send first information to the second device, where the first information is used to request evaluation data of N operation tasks in the first operation link, and N is a positive integer.

In one possible design, the communication module is further configured to receive second information from the third device. The second information is used for requesting to evaluate the network autonomy level of the first operation link, and the second information comprises the first information. Optionally, the second information includes content in the first information, but the signaling format of the first information is different from that of the second information; alternatively, the first information is encapsulated in the second information, and the first information is a part of the second information.

In one possible design, the evaluation data of the N operation and maintenance tasks of the first information request includes one or more of the following: the information of the N operation and maintenance tasks, the execution mode of the N operation and maintenance tasks and the execution effect of the N operation and maintenance tasks. The design clearly shows the type of the evaluation data, is convenient for the second device to quickly and effectively collect the required evaluation data, and is beneficial to improving the evaluation efficiency.

In one possible design, the communication module is further configured to receive third information from the second device, the third information being configured to indicate one or more of: and the processing of the N operation and maintenance tasks is completed, the execution mode of the N operation and maintenance tasks is completed, and the execution effect of part or all of the N operation and maintenance tasks is achieved.

In a fifth aspect, an embodiment of the present application provides an evaluation device for an autonomous level of an operation and maintenance network, where the device may be a second device, or may be a module or a chip in the second device, or may be a device that can be used in a matching manner with the second device. In one design, the apparatus may include modules corresponding to the methods/operations/steps/actions described in the second aspect, where the modules may be implemented by hardware circuits, software, or a combination of hardware circuits and software. In one design, the apparatus may include a processing module and a communication module. Comprising the following steps:

The communication module is used for receiving first information from the first device, wherein the first information is used for requesting evaluation data of N operation and maintenance tasks in a first operation and maintenance operation link, and N is a positive integer.

And the processing module is used for sending third information to the first device through the communication module according to the first information, wherein the third information is used for indicating the evaluation data of the N operation and maintenance tasks.

In a sixth aspect, an embodiment of the present application provides an apparatus for evaluating an autonomous level of an operation and maintenance network, where the apparatus includes a processor configured to implement the method described in the first aspect. The processor is coupled to a memory for storing instructions and data, and the processor, when executing the instructions stored in the memory, may implement the method described in the first aspect. Optionally, the evaluation device of the autonomous level of the operation and maintenance network may further include a memory; the apparatus for evaluating an autonomous level of an operation and maintenance network may further comprise a communication interface for the apparatus to communicate with other devices, which may be, for example, a transceiver, a circuit, a bus, a module, a pin or other type of communication interface.

In one possible design, the apparatus includes:

and the memory is used for storing the instructions.

And the processor is used for acquiring the evaluation data of N operation and maintenance tasks in the first operation and maintenance operation link from the second device, wherein N is a positive integer.

The communication interface is used for sending the evaluation data of the N operation and maintenance tasks to a third device, wherein the evaluation data of the N operation and maintenance tasks are used for determining the network autonomy level of the first operation and maintenance operation link; or,

The processor is further configured to determine, by the first device, a network autonomous level of the first operation link according to the evaluation data of the N operation tasks.

In a seventh aspect, an embodiment of the present application provides an apparatus for evaluating an autonomous level of an operation and maintenance network, where the apparatus includes a processor configured to implement the method described in the second aspect. The processor is coupled to a memory for storing instructions and data, and the processor, when executing the instructions stored in the memory, is capable of implementing the method described in the second aspect. Optionally, the evaluation device of the autonomous level of the operation and maintenance network may further include a memory; the apparatus for evaluating an autonomous level of an operation and maintenance network may further comprise a communication interface for the apparatus to communicate with other devices, which may be, for example, a transceiver, a circuit, a bus, a module, a pin or other type of communication interface.

In one possible design, the apparatus includes:

and the memory is used for storing the instructions.

The communication interface is used for receiving first information from the first device, wherein the first information is used for requesting evaluation data of N operation and maintenance tasks in a first operation and maintenance operation link, and N is a positive integer.

And the processor is used for sending third information to the first device through the communication module according to the first information, wherein the third information is used for indicating the evaluation data of the N operation and maintenance tasks.

In an eighth aspect, an embodiment of the present application provides an evaluation system for an autonomous level of an operation and maintenance network, where the system includes a first device and a second device. Optionally, the system may further comprise a third means. Wherein:

the second device is configured to send evaluation data of N operation and maintenance tasks in the first operation and maintenance operation link to the first device, where N is a positive integer.

The first device is used for determining the network autonomy level of the first operation link according to the evaluation data of the N operation tasks; or,

the first device is used for sending the evaluation data of the N operation and maintenance tasks to the third device;

and the third device is used for determining the network autonomy level of the first operation link according to the evaluation data of the N operation tasks.

In a ninth aspect, an embodiment of the present application provides an evaluation system for an autonomous level of an operation and maintenance network, where the system includes an evaluation device for an autonomous level of an operation and maintenance network as described in the fourth aspect or the sixth aspect, and an evaluation device for an autonomous level of an operation and maintenance network as described in the fifth aspect or the seventh aspect.

In a tenth aspect, embodiments of the present application further provide a computer program, which when run on a computer causes the computer to perform the method provided in any one of the first to second aspects above.

In an eleventh aspect, embodiments of the present application also provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method provided in any one of the first to second aspects above.

In a twelfth aspect, embodiments of the present application further provide a computer-readable storage medium having stored therein a computer program or instructions which, when run on a computer, cause the computer to perform the method provided in any one of the first to second aspects above.

In a thirteenth aspect, embodiments of the present application further provide a chip for reading a computer program stored in a memory, performing the method provided in any of the first to second aspects, or comprising circuitry for performing the method provided in any of the first to second aspects.

In a fourteenth aspect, embodiments of the present application further provide a chip system, which includes a processor for supporting a device to implement the method provided in any one of the first aspect to the second aspect. In one possible design, the system-on-chip further includes a memory for storing programs and data necessary for the device. The chip system may be formed of a chip or may include a chip and other discrete devices.

Effects of the solutions provided in any of the above second aspect to fourteenth aspect, reference may be made to the corresponding description in the first aspect.

Drawings

Fig. 1 is a schematic structural diagram of an operation and maintenance network according to an embodiment of the present application;

FIG. 2 is a schematic diagram of operation and maintenance operation link division relationship based on a technical domain and a service domain;

fig. 3 is a flowchart of an evaluation method for an autonomous level of an operation and maintenance network according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of an operation and maintenance network according to an embodiment of the present application;

FIG. 5 is a schematic flow chart of an evaluation method for network coverage optimization according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of an operation and maintenance network according to an embodiment of the present application;

FIG. 7 is a schematic flow chart of an evaluation method for network coverage optimization according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of an operation and maintenance network according to an embodiment of the present application;

FIG. 9 is a schematic flow chart of an evaluation method for network coverage optimization according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of an evaluation device for an autonomous level of an operation and maintenance network according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of an evaluation device for an autonomous level of an operation and maintenance network according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include, for example, "one or more" such forms of expression, unless the context clearly indicates to the contrary. It should also be understood that in embodiments of the present application, "one or more" means one, two, or more than two; "and/or", describes an association relationship of the association object, indicating that three relationships may exist; for example, a and/or B may represent: a alone, a and B together, and B alone, wherein A, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship.

Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

The plurality of the embodiments of the present application refers to greater than or equal to two. It should be noted that, in the description of the embodiments of the present application, the terms "first," "second," and the like are used for distinguishing between the descriptions and not necessarily for indicating or implying a relative importance, or alternatively, for indicating or implying a sequential order.

Technical terms related to the embodiments of the present application will be first described below.

(1) Operation and maintenance network

An operation and maintenance network may also be referred to as an operation and maintenance network system, referring to a system that manages and maintains a communication network. The operation and maintenance network according to the embodiment of the present application may be AN autonomous network (autonomous network, AN), where the autonomous network refers to a telecommunication system (including a telecommunication network and AN operation management system thereof) that implements self-management (telecommunication system (including management system and network) with autonomy capabilities which is able to be governed by itself with minimal to no human intervention) through autonomous capability with as little manual intervention as possible.

In particular, the operation and maintenance network can be divided into two functions, such as a network management function and a network element management function. Or it may be described that the operation and maintenance network includes two types of functional systems, one type of functional system provides management functions at the network level, and the other type of functional system provides management functions at the network element level. The management functions of the network layer may include monitoring, analyzing, configuring, maintaining, etc. the network of the operator. The management function of the network element layer mainly focuses on the network element management in the communication network, and the end-to-end management and maintenance between the network element and the network are realized.

Alternatively, the system for providing the management function at the network level in the operation and maintenance network may be a network management system (network management system, NMS) or other system, and the system for providing the management function at the network element level may be a network element management system (network element management system, EMS) or other system.

As an example, an operation and maintenance network 100 is illustrated in fig. 1, and the operation and maintenance network 100 includes a network management system NMS and a network element management system EMS. The NMS may provide a communication interface for the user, for example, the NMS may receive an instruction or command of a network administrator, perform management operations such as monitoring, analysis, configuration, maintenance, etc. on the network of the operator, and may communicate with other systems outside the network of the operator, for example. The EMS may communicate with Network Elements (NEs) in the communication network, such as collect data of the network elements in the communication network and send the collected data to the NMS. Alternatively, the EMS may establish a connection with the NMS through a northbound interface (northbound interface, NBI), i.e. the EMS may send data to the NMS through the northbound interface NBI. One or more devices providing network management functions may be included in the NMS and one or more devices providing network element management functions may be included in the EMS. In one possible design, a network element in communication with the EMS may also be included in the operation and maintenance network.

It will be appreciated that the communication network may be of some kind, for example, a third generation (3 ^th generation, 3G) communication network, fourth generation (4 ^th generation, 4G) communication networks (e.g., long term evolution (long term evolution, LTE) networks), fifth generation (5 ^th generation, 5G) communication network, worldwide interoperability for microwave access (worldwide interoperability for microwave access, wiMAX) or wireless local area network (wireless local area network, WLAN) network, or a converged network of multiple networks, or future communication networks, such as a 6G communication network, etc. In particular, the communication network may also be a network over a technical domain, such as a digital energy network, a data center network, a transport network, a bearer network, a core network, a fixed access network, a radio access network, etc. The communication network may also be a network where an operator provides a certain service, where different services correspond to different areas or types, such as an enterprise campus network, a mobile network (mobile 2B) for enterprise users, a mobile network (mobile 2C) for personal users, a private network for administrative offices, a home broadband network, etc.

The network element in the communication network may be an entity providing a network service, may be a core network element, an access network element, etc. For example, core network elements may include, but are not limited to, access and mobility management function (access and mobility management function, AMF) entities, session management function (session management function, SMF) entities, policy control function (policy control function, PCF) entities, network data analysis function (network data analysis function, NWDAF) entities, network storage function (network repository function, NRF), gateways, and the like. The access network elements may include, but are not limited to, various types of base stations (e.g., next generation base stations (generation node B, gNB), evolved node B (eNB), centralized control units (central unit control panel, CU-CP), centralized Units (CU), distributed Units (DUs), centralized user plane units (central unit user panel, CU-UP), and the like.

(2) Operation and maintenance operation link and operation and maintenance task

The operation and maintenance operation link refers to a main production activity for operating and maintaining the communication network, and the operation and maintenance operation link defined by the related protocol comprises: network planning design, network deployment, network monitoring and barrier removal, service distribution, network change, network optimization (such as optimizing adjustment parameters) and the like. The operation link may refer to a main activity of an operation full life cycle in a network on a certain technical domain and/or a network on a certain service domain, corresponding to a communication network division situation for performing network operation and maintenance. Or may also be described as: the operation and maintenance operation links are for one or more network dimensions, which may correspond to a technical domain or a business domain. As shown in fig. 2, an operation and maintenance operation link dividing relationship based on a technical domain and a service domain is illustrated, taking the operation and maintenance operation link as an example for the technical domain and the service domain, the technical domain is a carrier network, the service domain is a private line of a government enterprise, and the operation and maintenance link of "network optimization" is specifically the network optimization of the carrier network for the private line of the government enterprise.

An operation and maintenance operation link corresponds to a workflow, and the workflow of an operation and maintenance operation link may be composed of one or more operation and maintenance tasks, or may be described as an operation and maintenance operation link may include one or more operation and maintenance tasks. The operation and maintenance Task can be denoted as Task.

By way of example, the following is an illustration of some of the aforementioned operation and maintenance links:

-network planning design: and carrying out the process of designing and planning for the newly built network or the modified network according to business, market, product and customer service demands and development expectations thereof.

-network deployment: and according to the network planning result, performing the processes of installing, configuring, activating and verifying the network so as to enable the network to have the opening condition.

-network maintenance: in order to ensure that the network is in a normal running state, the related information such as the network state and the like are monitored, the problems are analyzed and the problems are solved. Such as monitoring for faults, troubleshooting faults, and the like.

Network optimization: in order to improve network performance or communication service experience, relevant information such as network performance indexes is monitored and analyzed, and performance optimization measures such as network resources, parameter configuration adjustment and the like are adopted.

Taking the above "network optimization" as an example, specifically, network coverage optimization, the operation links corresponding to the network coverage optimization may include the following operation tasks: control information generation, intention evaluation, data collection, problem identification (or anomaly identification), degradation prediction, problem delimitation, root cause analysis, optimization scheme generation, optimization scheme evaluation and decision making and optimization scheme issuing.

Wherein, the control information generation (coverage optimization policies generation and determination) mainly comprises the generation and determination of network coverage optimization related policy control information. Intent assessment (coverage optimization intent evaluation), which refers primarily to assessing whether network coverage optimization achieves a target intent, which refers to the desire for requirements, goals, and constraints of a particular service or network management workflow. The data collection (coverage related information collection) mainly includes collecting relevant data for network coverage optimization. Problem identification (coverage issues identification) mainly includes analyzing the performance of network coverage and determining problems with network coverage. Degradation prediction (coverage deterioration prediction) consists essentially of analyzing current and historical network coverage performance, predicting future network coverage performance trends, and identifying in advance problems with potential network coverage performance degradation. Problem definition (coverage issues demarcation) mainly includes analyzing network coverage problems and identifying categories of network coverage problems. Root cause analysis (coverage issue root cause analysis) primarily refers to analyzing and determining root causes that lead to network coverage problems. The optimization scheme generation (coverage adjustment solutions analysis) mainly includes generating a solution to the network coverage problem that can be used to address the determined or predicted network coverage problem. The optimization scheme evaluation and decision (coverage adjustment solutions evaluation and determination) mainly includes evaluation and determination of the network coverage problem adjustment scheme.

It should be noted that, the above dividing manner of the operation and maintenance operation links and the operation and maintenance tasks included in the operation and maintenance operation links are only exemplary, which is not limited in this embodiment of the present application.

(3) Network autonomy level

Network autonomy level refers to an autonomy capability level (autonomous network level) of a network. In the operation and maintenance network, the network autonomous level can be evaluated for each operation and maintenance operation link respectively, and one operation and maintenance operation link in the operation and maintenance network can be understood to be an evaluation object. One operation Task (Task) in the operation and maintenance operation link can be used as the minimum evaluation granularity in the evaluation process for the operation and maintenance operation link, and the network autonomy level of one operation and maintenance operation link can be determined by the autonomy capability of one or more operation and maintenance tasks included in the network autonomy level.

(4) Man-machine division state

The man-machine division state is used for reflecting the participation degree or the proportion of the manual mode and the system automatic mode in the operation and maintenance task completion process. The man-machine division state can also be described as an execution mode of the operation and maintenance task and can be used for determining the network autonomous capacity corresponding to the operation and maintenance task. From a fully manually completed state to a fully automatically completed state by the system, various human-machine division states can be divided.

Illustratively, the human-machine division state may include:

and (3) manual work: the operation and maintenance tasks are executed by people, namely, are completed by a manual mode;

manual binding system (manual + system): the operation and maintenance tasks are jointly executed by people and the system, namely, the operation and maintenance tasks are completed by a manual mode and an automatic mode of the system. Optionally, the "manual+system" state can be further subdivided into:

-a manual assistance system: the operation and maintenance task takes the manual mode as the leading mode and is completed by the system; further, according to the participation degree or proportion of the labor and the system in completing the operation and maintenance task, the human-machine division state of the 'manual auxiliary system' can be further subdivided, for example, the human-machine division state can be further subdivided into '60% manual +40% system', '80 manual +20% system', '90% manual +10% system' and the like;

-system assisted manual: the operation and maintenance task takes the system automatic mode as the leading and is completed manually; further, according to the participation degree or proportion of the labor and the system in completing the operation and maintenance task, the human-machine division state of the system auxiliary labor can be further subdivided, for example, the system auxiliary labor can be further subdivided into 10 percent labor +90 percent system, 20 percent labor +80 percent system, 40 percent labor +60 percent system and the like;

The system comprises: the operation and maintenance tasks are completed by the system in an automatic manner.

Different man-machine division states correspond to different network autonomy capabilities. For example, if the man-machine division state includes: the 3 man-machine division states are ordered as follows according to the network autonomy capability from high to low: system, manual + system, system.

It should be noted that, the above division manner of the man-machine division state is only exemplary, and the embodiments of the present application do not limit this.

Aiming at the problem of poor reliability caused by the fact that an operation and maintenance network provider provides an operation and maintenance network autonomous level evaluation method, the embodiment of the application provides an operation and maintenance network autonomous level evaluation method, evaluation data defining each operation and maintenance task can be transmitted among a plurality of devices/apparatuses, so that an evaluation scheme based on the evaluation data of each operation and maintenance task is transparent, and the reliability of operation and maintenance network autonomous level evaluation can be improved. The embodiments of the present application are further described in detail below.

Fig. 3 illustrates an evaluation method for an autonomous level of an operation and maintenance network according to an embodiment of the present application, where the method can be applied to the foregoing operation and maintenance network, for example, the operation and maintenance network illustrated in fig. 1.

Specifically, the method mainly comprises the following steps:

S301, the first device acquires evaluation data of N operation and maintenance tasks in a first operation and maintenance operation link from the second device.

Wherein N is a positive integer. The evaluation data of the N operation and maintenance tasks are used for evaluating the network autonomy level of the first operation and maintenance operation link. Alternatively, N may be the number of all the operation tasks included in the first operation link, or N may be the number of part of the operation tasks included in the first operation link, where the proportion of the part of the operation tasks to all the operation tasks is greater than a set threshold, such as 80%.

The first device may be understood as a device for evaluating the autonomous level of the decision network, may be one of an NMS or an EMS in the operation and maintenance network as illustrated in fig. 1, or may be a device having an autonomous level evaluation function of the decision network, which is separately deployed in the operation and maintenance network.

The second apparatus may include an execution entity for processing the N operation and maintenance tasks, and the number of the execution entity may be one or more. The executing entity may be an apparatus or a network element entity with a network element management function in the operation and maintenance network. For example, one implementation entity is one device in the EMS in the schematic operation and maintenance network of fig. 1, or may be understood as follows: the second device includes one or more devices in the EMS. Alternatively, in the embodiment of the present application, one operation and maintenance task is processed by one executing entity, but one executing entity may process one or more operation and maintenance tasks.

The apparatus in the embodiments of the present application may also be alternatively described as a system, a subsystem, a device, a module, or a unit, etc., for example, the first apparatus may also be referred to as a first system, a first subsystem, a first module, a first device, a first unit, etc., which is not limited by the embodiments of the present application.

In an alternative embodiment, the first device is not comprised by the second device. The first device may send first information to the second device, the first information being used to request evaluation data of N operation and maintenance tasks in the first operation and maintenance work link. Accordingly, the second device may send third information to the first device, the third information being used to indicate evaluation data of the N operation and maintenance tasks in the first operation and maintenance job link.

The definition of the evaluation data can be understood with reference to the following two examples.

Example 1, the type of evaluation data may be configured in a predefined manner such that the understanding of the evaluation data by the various devices in the operation and maintenance network remains consistent. For example, for a particular operation and maintenance task, the evaluation data of the operation and maintenance task may include one or more of an identification of the operation and maintenance task, an execution mode of the operation and maintenance task, or a human-machine-division status of the operation and maintenance task, and an execution effect of the operation and maintenance task. Wherein the identification of an operation and maintenance task is used for indicating the information of the operation and maintenance task.

Accordingly, the second device may indicate in the third information one or more of the following according to the type of the predefined evaluation data: and the processing of the N operation and maintenance tasks is completed, the execution mode of the N operation and maintenance tasks is completed, and the execution effect of the N operation and maintenance tasks is achieved.

Example 2, the first device may indicate in the first information the type of evaluation data of N operation and maintenance tasks requested by the first device (the type may alternatively be described as a definition), for example, the evaluation data of any one of the N operation and maintenance tasks requested by the first information includes one or more of the following data: the information of any operation and maintenance task, the execution mode of any operation and maintenance task, the execution effect of any operation and maintenance task; alternatively, it may alternatively be described as: the evaluation data of the ith operation and maintenance task in the N operation and maintenance tasks of the first information request comprises one or more of the following data: information of the ith operation and maintenance task, an execution mode of the ith operation and maintenance task and an execution effect of the ith operation and maintenance task. I is an integer from 1 to N. The data included by different operation tasks in the N operation and maintenance may be the same or different. The second device may determine which data the evaluation data required by the first device specifically comprises, based on the indication in the first information.

Alternatively, the indication in the first information may be represented as: the first information may include a task set list including information of N operation and maintenance tasks, such as identification of the N operation and maintenance tasks. The first information may further include a type of evaluation data, such as an execution mode of the operation and maintenance task, an execution effect of the operation and maintenance task, and the like.

Optionally, when the evaluation data required by the first device includes an execution mode of the operation and maintenance task, the first device may further indicate a candidate mode of the execution mode in the first information, that is, provide a selection range of the execution mode. For example, the first device may include first indication information for indicating a candidate pattern of the execution pattern in the first information. As an example, the candidate modes corresponding to the execution mode include: a manual execution mode (corresponding to the aforementioned human-machine division state is manual), a manual and system common execution mode (corresponding to the human-machine division state is a human-machine combination system), and a system execution mode (corresponding to the human-machine division state is a system). Of course, alternatively, the first device may also indicate the subdivision range of the manual combining system in the first information, and this design may be described with reference to the foregoing terminology for the man-machine division status, which is not repeated in the embodiments of the present application.

Optionally, when the evaluation data required by the first device includes the execution effect of the operation and maintenance task, the first device may further indicate, in the first information, index information for measuring the execution effect. It can be understood that the index information is related to the operation and maintenance tasks, and the index information corresponding to different operation and maintenance tasks is different. Taking the operation and maintenance link of 'network coverage optimization' as an example, wherein index information of execution effects of problem delimitation, problem identification and root cause analysis can comprise accuracy and self-intelligence rate (Autonomy ratio); the index information of the execution effect of the degradation prediction may include a false alarm rate (false alarm ratio), a omission rate (missing alarm ratio), a self-intelligence rate, and the like. The self-intelligence rate is used for indicating the automatic completion proportion of the operation and maintenance tasks by the system, for example, for root cause analysis, the self-intelligence rate of the root cause analysis can indicate the proportion of the number of cells for which the root cause analysis is completed by the system to the total number of cells subjected to the root cause analysis.

The third information is used for indicating one or more of the following data corresponding to the type of the evaluation data indicated in the first information: and the processing of the N operation and maintenance tasks is completed, the execution mode of the N operation and maintenance tasks is completed, and the execution effect of part or all of the N operation and maintenance tasks is achieved. It is understood that the type of the evaluation data indicated in the third information includes the type of the evaluation data indicated in the first information.

The following describes in detail a specific embodiment in which the first device transmits the first information to the second device.

In one case, the first device is one device in the NMS and the second device includes one or more devices in the EMS.

When the second device includes 1 device in the EMS, the first device may transmit the first information to the second device through one signaling.

When the second device includes a plurality of devices in the EMS, it can be understood that each of the plurality of devices is used to process a portion of the N operation and maintenance tasks. The first device may transmit the first information to 1 device of the plurality of devices through one signaling. The 1 devices are recorded as a fourth device, and the fourth device can determine the execution entity corresponding to each operation and maintenance task in the N operation and maintenance tasks according to the first information. The fourth means may request evaluation data of the part of the operation and maintenance task which it processes from the remaining execution entities other than itself. Taking the fifth device for processing N1 operation and maintenance tasks as an example: the fourth device may send fourth information to the fifth device, the fourth information being used to request evaluation data of the N1 operation and maintenance tasks. Alternatively, the fourth information may include a part of the content in the first information, for example, the first information includes a task set list, and when the task set list includes information of N operation and maintenance tasks, the fourth information may include a part of the content in the task set list, that is, information of N1 operation and maintenance tasks.

In a second case, the first device is one device of the EMS, the second device includes one or more devices of the EMS, and the first device is not included in the second device.

When the second device includes a plurality of devices in the EMS, it can be understood that each of the plurality of devices is used to process a portion of the N operation and maintenance tasks. The first device may transmit the first information to 1 device of the plurality of devices through one signaling. The 1 devices are recorded as a fourth device, and the fourth device can determine the execution entity corresponding to each operation and maintenance task in the N operation and maintenance tasks according to the first information. The fourth means may request the evaluation data of the part of the operation and maintenance tasks which it processes from the remaining execution entities other than itself; or,

when the second device includes a plurality of devices in the EMS, it can be understood that each of the plurality of devices is used to process a portion of the N operation and maintenance tasks. The first device may send the first information to the plurality of devices through a plurality of signaling. The plurality of signaling corresponds to the plurality of devices one by one, and the first device may send signaling corresponding to the 1 device to 1 device in the plurality of devices, where the signaling corresponding to the 1 device is used to request evaluation data of a part of operation and maintenance tasks that can be processed by the 1 device. Optionally, the first device may indicate, in the signaling corresponding to the 1 device, the type of the evaluation data of the corresponding partial operation and maintenance tasks, corresponding to the type of the evaluation data including N operation and maintenance tasks in the first information. A detailed description of a specific embodiment of the second device transmitting the third information to the first device will be provided below.

When the second device includes 1 device in the EMS, the second device transmits third information to the first device through one signaling.

When the second device includes a plurality of devices in the EMS, the third information may be composed of a plurality of signaling, which corresponds to the plurality of devices one by one. 1 device of the plurality of devices sends signaling corresponding to the 1 device to the first device, wherein the signaling corresponding to the 1 device is used for indicating evaluation data of part of operation and maintenance tasks processed by the 1 device. The first device determines third information based on the plurality of received signaling.

Taking the example that the fifth device included in the second device is used for processing N1 operation and maintenance tasks in N operation and maintenance tasks: the fifth device transmits fifth information indicating evaluation data of the N1 operation and maintenance tasks to the first device. The fifth information may also be described as signaling corresponding to the fifth device. Optionally, the fifth information may include part of the content in the third information, for example, the fifth information is used to indicate one or more of the following: the processing of the N1 operation and maintenance tasks is completed, the execution mode of the N1 operation and maintenance tasks is completed, and the execution effect of the N1 operation and maintenance tasks is achieved.

In another alternative embodiment, the second device includes a plurality of devices in the EMS and the first device is included in the second device. The first device may determine evaluation data of a part of the operation and maintenance tasks that can be processed by itself, and the first device may request the evaluation data of the corresponding operation and maintenance tasks from the rest of the second devices except the first device. The first device requests and obtains the evaluation data corresponding to the operation task from the other devices, and the fourth device may request the evaluation data from other execution entities except for the first device in the foregoing case, which is not described in detail in the embodiment of the present application.

Alternatively, the first device may actively obtain the evaluation data of the N operation and maintenance tasks from the second device. Alternatively, the first device may obtain the evaluation data of the N operation and maintenance tasks from the second device under the trigger of the third device. As an example, fig. 3 also illustrates an optional step S300 before S301: the third device sends second information to the first device, the second information being used to request an assessment of the network autonomy level of the first operation and maintenance link.

Alternatively, the third device may indicate the required evaluation data in the second information, for example, the second information includes second indication information, where the second indication information is used to indicate the evaluation data required for evaluating the network autonomous level of the first operation and maintenance link. For example, the evaluation data indicated by the second indication information may include one or more of the following: information for indicating the N operation and maintenance tasks, execution modes of the N operation and maintenance tasks, and execution effects of the N operation and maintenance tasks. Further optionally, the third device may further include the aforementioned first indication information, a subdivision range of the manual combination system, and the like in the second information.

Alternatively, when the first device is one of the NMSs, it may perform: the first device actively acquires the evaluation data of N operation and maintenance tasks from the second device; when the first device is one of the EMS devices, it may perform: the first device acquires the evaluation data of N operation and maintenance tasks from the second device under the triggering of the third device.

Optionally, the third means may provide a communication interface for the user, capable of providing the user with relevant data of the operation and maintenance network. When the first device is one of the EMS devices, the third device may be one of the NMSs in the operation and maintenance network illustrated in fig. 1, or a third party evaluation system outside the operation and maintenance network. The first device may provide the evaluation data or the evaluation results of the N operation and maintenance tasks to users other than the operation and maintenance network provider through the third device.

Further, the method further includes S302A or S302B-S303. For example, when the first device is one of NMS, S302A is performed; when the first device is one of the EMS, S302B and S303 are performed.

S302A, the first device determines the network autonomy level of the first operation link according to the evaluation data of the N operation tasks.

It may be understood that the network autonomous level of the first operation and maintenance operation link belongs to an evaluation result of the first device for evaluating the first operation and maintenance operation link, and the evaluation result may also include other contents, which is not limited in the embodiment of the present application.

In an optional implementation manner, the first device may determine an automation level of each of the N operation and maintenance tasks according to an execution mode of the N operation and maintenance tasks; and further, calculating the network autonomy level of the first operation and maintenance operation link according to the automation capacity and the corresponding weight of each operation and maintenance task by using a weighted average algorithm. For example, it can be expressed as the following formula:

level＝∑ _i T _i ×w _i ；

Wherein T is _i Is the automation of the ith operation and maintenance task in N operation and maintenance tasksThe chemical capabilities, the automation capability of an operation and maintenance task can be determined by the execution mode and/or the execution effect of the operation and maintenance task. The execution effect is used for indicating the result or performance of the completion of the operation and maintenance task, taking the root cause analysis as an example, the execution effect of the root cause analysis can be the accuracy of the root cause analysis. w (w) _i Is the weight corresponding to the ith operation and maintenance task, w _i The weights of different operation and maintenance tasks can be the same or different, and the embodiment of the application is not limited. level is the level of network autonomy of the evaluation object (i.e., the first operation link). i is a positive integer from 1 to N.

In another optional implementation manner, the first device may determine the network autonomy level of the first operation and maintenance operation link according to the type of the first operation and maintenance operation link and the execution mode of the N operation and maintenance tasks. Taking the type of the first operation link as network coverage optimization as an example, the first operation link may include operation tasks (tasks) a-J, and the human-machine division mode (i.e., execution mode) of each operation Task may be a human (P), a human-machine combination system (P/S) or a system (P). As shown in table 1, the network autonomous level of the "network coverage optimization" may be one of levels (level) 1 to 5, and the man-machine division modes corresponding to tasks a to J in different levels are not completely the same. Specifically, the network autonomous level of the "network coverage optimization" may be determined by the human-machine division mode of the operation and maintenance tasks a to J, for example, when the first device determines that the obtained human-machine division state of the operation and maintenance tasks a to J accords with the human-machine division state of the operation and maintenance tasks a to J in the column of the level 1 indicated in table 1, the network autonomous level of the operation and maintenance operation link of the "network coverage optimization" may be determined to be the level 1.

TABLE 1

S302B, the first device sends evaluation data of the N operation and maintenance tasks to a third device.

S303, the third device determines the network autonomy level of the first operation and maintenance link according to the evaluation data of the N operation and maintenance tasks.

Specifically, this step may be implemented with reference to S302A, which is not described in detail in the embodiments of the present application.

In the evaluation method illustrated in fig. 3, the device for deployment decision evaluation communicates with the device for processing the operation and maintenance tasks, and evaluation data of a plurality of operation and maintenance tasks are collected, so that the network autonomous level of the operation and maintenance operation link is determined based on the evaluation data of the plurality of operation and maintenance tasks, an objective and transparent evaluation scheme can be realized, and the reliability of an evaluation result is improved.

For easy understanding, the method for evaluating the autonomous level of the operation and maintenance network provided in fig. 3 will be further described below by taking the first operation and maintenance operation link as an example of "network coverage optimization".

Example one

As shown in fig. 4, an operation and maintenance network architecture is illustrated, and based on the operation and maintenance network illustrated in fig. 1, it is further illustrated that an NMS system in the operation and maintenance network includes a consumer (consumer) system intended to drive a management function (intent driven management function, IDMF). The EMS system in the operation and maintenance network includes an IDMF producer (producer) device and a management function (management function, mnF) device that interacts with the IDMF device. MnF devices include data collection and coordination function (data collection coordination function, DCCF) devices, management data analysis function (management data analytics function, MDAF) devices, and the like.

The IDMF consumer system, the IDMF producer device, the DCCF device, and the MDAF device may be simply referred to as IDMF consumer, IDMF producer, DCCF, and MDAF, respectively. For ease of understanding, the functionality of IDMF consumer, IDMF, DCCF, and MDAF in network coverage optimization will be described first.

-IDMF consumer: a user's instructions may be received responsible for issuing an intent or intent request to the IDMF. For example, the IDMF consumer may issue a network coverage optimization creation intent, a network coverage optimization rating intent, to the IDMF. The IDMF consumer may also be used for decision making evaluation, for example, to summarize evaluation data of each operation and maintenance task in network coverage optimization, and to determine whether a certain operation and maintenance task is executed or not and the corresponding execution mode. Alternatively, a network autonomous level assessment (EVA) device may be deployed in the IDMF consumer, with decision assessment by the EVA.

-IDMF producer: responsible for intent translation, intent control, intent analysis and decision function. For example, the intent translation module is responsible for translating network coverage optimization intents and network coverage optimization hierarchical assessment intents issued by IDMF consumer into a series of policies, such as data collection policies, root cause analysis policies, and the like. The intent control function is responsible for sending control instructions to the MnF interacting with the IDMF (e.g., sending data collection instructions to the DCCF). The intent analysis and decision function is responsible for analyzing and deciding based on policies and data, such as root cause analysis, covering optimization scheme decisions. The intent analysis and decision function may be deployed inside the IDMF or may be accomplished by invoking MDAF. The IDMF producer may also be referred to simply as IDMF.

-DCCF: is responsible for data collection and coordination functions, such as collecting performance data and alarming data.

-MDAF: and is responsible for completing analysis tasks such as problem identification, root cause analysis, scheme generation and the like based on the acquired data.

Corresponding to the communication method described in fig. 3, the IDMF consumer may be understood as an example of the first apparatus; or the IDMF consumer deploys EVA, the EVA may be understood as an example of the first device. Accordingly, the IDMF producer, DCCF, MDAF are examples of the aforementioned second device, or may be described as the second device including the IDMF producer, DCCF, MDAF, and the like.

In the first example, the IDMF consumer acquires evaluation data of N operation and maintenance tasks in the network coverage optimization from the IDMF producer, the DCCF and the MDAF, and further determines a network autonomous level of the network coverage optimization according to the evaluation data of N operation and maintenance tasks in the network coverage optimization. Optionally, the value of N is the number of all operation and maintenance tasks in the network coverage optimization.

Specifically, the implementation may be referred to the scenario described in case one in S301. If the IDMF consumer sends first information to one of the IDMF producer, the DCCF and the MDAF; the device which receives the first information sends the evaluation data of the operation and maintenance tasks of the corresponding part of the signaling request to the rest of the execution entities; and further, the IDMF producer, the DCCF, and the MDAF send signaling to the IDMF consumer to indicate the evaluation data of the operation and maintenance tasks of the respective corresponding portions, and the IDMF consumer may obtain the third information according to the signaling from the IDMF producer, the DCCF, and the MDAF to the IDMF. The third information is used to indicate evaluation data of the N operation and maintenance tasks in the network coverage optimization.

As an example, fig. 5 illustrates a method for evaluating network coverage optimization, which is an example of the method illustrated in fig. 3, and may be applied to the operation and maintenance network illustrated in fig. 4. The method mainly comprises the following steps:

s501, the IDMF consumer sends first information to the IDMF producer.

Wherein, this step corresponds to the scheme described in the case one in S301, the IDMF consumer corresponds to the first device, and the IDMF producer corresponds to the fourth device. It will be appreciated that the first information herein is for requesting evaluation data for N operation and maintenance tasks in network coverage optimization. The first information may also be described as network coverage optimization rating evaluation intent.

Alternatively, when EVA is configured in the IDMF consumer, the IDMF consumer may send the first information to the IDMF producer through the EVA.

Optionally, the first information may include a coverage optimization evaluation indication, where the coverage optimization evaluation indication is used to indicate evaluation data required for evaluating the operation and maintenance link of network coverage optimization; alternatively, coverage optimization evaluation may be understood as evaluation data indicating N operation and maintenance tasks for indicating the first information request.

For example, the coverage optimization evaluation indication may include a Task (Task) set list indicating network coverage optimization to represent evaluation data that requires each operation and maintenance Task in the Task set list. The Task set list includes identification of N operation and maintenance tasks in network coverage optimization, for example, the N operation and maintenance tasks include intent translation, control information generation, intent evaluation, data collection, problem identification (or called anomaly identification), degradation prediction, problem delimitation, root cause analysis, optimization scheme generation, optimization scheme evaluation and decision, and optimization scheme issuing. The coverage optimization evaluation indication may also contain an identification of the Task execution mode (or Task man-machine division status) to indicate the execution mode of the operation and maintenance Task included in the evaluation data required for each operation and maintenance Task. Optionally, the coverage optimization evaluation indication may further include first indication information, that is, a candidate mode of the Task execution mode.

S502, the IDMF consumer sends network coverage optimization creation intention to the IDMF producer.

The network coverage optimization creation intent is used to request creation of the network coverage optimization operation link. Optionally, the network coverage optimization creation intention includes an indication of a network coverage optimization policy, such as an indication of a coverage optimization area, a coverage optimization target, and the like.

S503, the IDMF producer translates the intention of creating the network coverage optimization and the first information, generates a related strategy of the network coverage optimization, and establishes a corresponding relation between the operation and maintenance task of the network coverage optimization and the execution entity.

The relevant policies of network coverage optimization may include: coverage optimization data collection rules, coverage optimization problem identification policies, coverage optimization root cause analysis policies, and calculation formulas for achieving intent targets, and the like.

As an example, establishing a correspondence between the network coverage optimized operation and maintenance tasks and the execution entity may include: the data collection operation and maintenance task corresponds to DCCF, the problem identification operation and maintenance task corresponds to MDAF, the root cause analysis operation and maintenance task corresponds to MDAF, and the like.

S504, the IDMF producer transmits evaluation data intended for translation to the IDMF consumer.

In particular, the evaluation data of intent translation may include an intent translation completion notification, an execution mode of intent translation, and the like. The IDMF may determine the execution mode or the human division status of the intent translation based on whether human intervention is present in the intent translation process. Taking the man-machine division state as an example, the man-machine division state with intention translation when the person is in the middle is a man-machine combination system. The manual division state of the intent translation without manual intervention is a system.

Alternatively, when EVA is configured in the IDMF consumer, the IDMF producer specifically sends evaluation data for intended translation to the EVA.

Further, in the case one described in S301, the IDMF producer may be understood as an example of the fourth device, and the IDMF producer may request the evaluation data of the part of the operation and maintenance tasks processed by the IDMF producer from the rest of the execution entities (i.e., devices in the EMS) except for itself. Specifically, it can be understood with reference to the following steps S505 to S514:

s505, the IDMF producer sends a data collection request message to the DCCF.

In particular, the data collection request message may be used to request data collection of evaluation data of this operation and maintenance task. The data collection request message includes a portion of the first information, such as a Task list with only data collection identifiers and an identifier of a Task execution mode (or Task man-machine-division status). Alternatively, the portion of content may be described as a hierarchical assessment indication of data collection.

The data collection request message may also include a policy indication for the data collection operation task, such as indicating a monitoring area, a monitoring object, a monitoring target. The monitoring target may be understood as a target data type of data collection, including one or more of the following data: performance data (e.g., performance policies, minimization of drive tests (minimization of drive tests, MDT) data), alert data, cell configuration data, inventory data (e.g., equipment information, engineering parameters, etc.), environmental data (e.g., electronic maps, etc.).

S506, the DCCF collects data from the relevant network elements according to the data collection request message.

S507, the DCCF feeds back the collected data to the IDMF producer. For example, DCCF feeds back collected performance data, configuration data, etc. to IDMF producer.

In a possible case where the data collected by the DCCF from the relevant network element is part of the data requested by the IDMF producer, the DCCF may also request the data that cannot be received from the relevant network element from the outside. Optionally, the method further includes S508 to S509 after executing S507, and S508 to S509 are illustrated with a dotted line in fig. 5 as optional steps.

S508, the DCCF transmits an external data import request message to the IDMF consumer, and the message may include a data list, such as stock data, environment data, etc., for requesting external import.

S509, the IDMF consumer sends the externally imported data list to the IDMF producer.

S510, the DCCF sends the evaluation data of the data collection to the IDMF consumer.

Corresponding to the description in S505, the evaluation data of the data collection may include: and (5) notifying the completion of data collection and Task man-machine division state of data collection. It can be understood that if there is a flow of the DCCF requesting the external data in S508, the Task ergonomic state of data collection is an artificial combination system, and if there is no flow of the DCCF requesting the external data in S508, the Task ergonomic state of data collection is an artificial combination system.

Alternatively, when EVA is configured in the IDMF consumer, the DCCF specifically transmits the data-collected evaluation data to the EVA.

It is understood that the evaluation data collected by the data collection includes evaluation data of the operation and maintenance tasks processed by the DCCF among the N operation and maintenance tasks for network coverage optimization indicated by the third information.

S511, the IDMF producer sends a network coverage optimization request message to the MDAF.

The network coverage optimization request message may be used to request evaluation data for a portion of the operation and maintenance tasks handled by the MDAF in network coverage optimization. In a specific implementation, the network coverage optimization request message includes part of the content in the first information, for example, the network coverage optimization request message includes a Task list, the Task list includes part of the content of the Task set list in the first information, and the Task list includes an identifier of a part of the operation and maintenance tasks processed by the MDAF; the network coverage optimization request message also includes an identification of the Task execution mode (or Task man-machine division status). Exemplary, part of the operation and maintenance tasks of the MDAF process include problem identification, problem delimitation, degradation prediction, root cause analysis, and solution generation. Optionally, a portion of the content in the first information may also be described as a hierarchical assessment indication of a portion of the operation and maintenance tasks of the MDAF process.

Further, the network coverage optimization request message includes an indication of a network coverage optimization related policy.

S512, the MDAF processes a plurality of operation and maintenance tasks of network coverage optimization according to the network coverage optimization request message.

Specifically, the MDAF may accomplish the multiple operation and maintenance tasks involved in S511, such as problem identification, problem delimitation, degradation prediction, root cause analysis, and scheme generation, independently or with the aid of IDMF consumer.

Further, the MDAF may send the evaluation data of the plurality of operation and maintenance tasks processed by the MDAF consumer to the IDMF consumer, including steps S513 and S514, that is, the evaluation data of the plurality of operation and maintenance tasks processed by the MDAF includes the execution mode and the execution effect of the plurality of operation and maintenance tasks. It is understood that the evaluation data of the plurality of operation and maintenance tasks includes the evaluation data of the operation and maintenance tasks processed by the MDAF among the N operation and maintenance tasks of the network coverage optimization indicated by the third information.

S513, the MDAF sends the execution mode of the operation and maintenance tasks to the IDMF consumer.

Similarly, taking the problem identification in multiple operation and maintenance tasks as an example, the evaluation data of the problem identification may include a notification of completion of the problem identification and Task labor division status (e.g., system) of the problem identification. Optionally, when EVA is configured in the IDMF consumer, the MDAF specifically sends the EVA an execution mode of the plurality of operation and maintenance tasks.

S514, the MDAF sends the execution effect of the operation and maintenance tasks to the IDMF consumer. Examples of some of the effects of the execution are provided below: such as problem identification results (weak coverage), problem delimitation (cells where weak coverage occurs), root cause analysis results (power setting is unreasonable), scheme generation results (providing antenna power to set values). As an optional step S514 is illustrated in fig. 6 with a dashed line. Optionally, when the EVA is configured in the IDMF consumer, the MDAF specifically sends the EVA the execution effect of the plurality of operation and maintenance tasks.

S515, the IDMF producer performs optimization scheme evaluation and decision.

Specifically, the IDMF producer can complete the evaluation and decision of the optimization scheme independently or with the aid of the IDMF consumer, such as analyzing the coverage optimization effect of the optimization scheme, conflicts with other intents, and the like.

S516, the IDMF producer sends evaluation data of the optimization scheme evaluation and decision to the IDMF consumer.

Wherein, the evaluation data of the optimization scheme evaluation and decision may include: completion notification of the optimization scheme evaluation and decision, task human division status of the optimization scheme evaluation and decision (e.g. system without human intervention or human combination system with human intervention). It is understood that the evaluation data of the optimization scheme evaluation and decision includes the evaluation data of the operation and maintenance tasks processed by the IDMF producer among the N operation and maintenance tasks of the network coverage optimization indicated by the third information.

Optionally, when EVA is configured in the IDMF consumer, the IDMF producer specifically sends evaluation data of the optimization scheme evaluation and decision to the EVA.

S517, the IDMF producer issues the optimization scheme to the relevant network elements. Specifically, the IDMF producer may indicate a network element list corresponding to the optimization scheme and an operation list of the optimization scheme to the relevant network elements.

S518, the IDMF producer sends evaluation data issued by the optimization scheme to the IDMF consumer. The evaluation data issued by the optimization scheme comprises a completion notification issued by the optimization scheme and a Task human-machine division state issued by the optimization scheme (such as a system when no human intervention exists or a human-machine combination system when human intervention exists).

S519, IDMF producer performs intention evaluation.

In particular, the IDMF producer may determine whether to reach a network coverage optimization hierarchy assessment intent through an intent assessment, which may also be described as an intent reaching assessment. Alternatively, the IDMF producer may perform the intent achievement evaluation independently or with the aid of an IDMF consumer, such as evaluating whether the intent is achieved, the specifics of the intent being achieved, and the like.

S520, the IDMF producer sends evaluation data of the intention evaluation to the IDMF consumer.

The evaluation data of the intention evaluation comprise a completion notification issued by the optimization scheme and a Task man-machine division state of the intention evaluation.

Alternatively, when EVA is configured in the IDMF consumer, the IDMF producer specifically transmits evaluation data for the intended evaluation to the EVA.

It is understood that the evaluation data of the intention evaluation includes evaluation data of the operation and maintenance tasks processed by the IDMF producer among the N operation and maintenance tasks of the network coverage optimization indicated by the third information.

S521, the IDMF producer feeds back the result of the intention evaluation to the IDMF consumer. Wherein the result of the intent assessment may be used to indicate the specifics of the intent to complete. Alternatively, the result of the intention evaluation may also be used as evaluation data for evaluating the network autonomy level of the network coverage optimization.

It is understood that the content indicated by the third information in this example one includes the evaluation data referred to in S504, S510, S513, S514, S516, S518, S520, and S521, corresponding to the second apparatus described in S301 transmitting the third information to the first apparatus.

S522, the IDMF consumer determines the network autonomy level of the network coverage optimization according to the received evaluation data of each operation and maintenance task in the network coverage optimization.

Specifically, this step corresponds to that described in S302A: the first device determines the network autonomy level of the first operation and maintenance link according to the evaluation data of the N operation and maintenance tasks of the first operation and maintenance link. The step S522 may be implemented by referring to the determination scheme of the network autonomous level in the first operation and maintenance link described in S302A, which is not described in detail in the embodiment of the present application.

Alternatively, when EVA is configured in the IDMF consumer, the level of network autonomy for network coverage optimization may be determined by EVA. Further, EVA stores network autonomous levels of network coverage optimization in IDMF consumer.

In the first example, the IDMF consumer collects the evaluation data of each Task, and supports the external manufacturer to evaluate the network autonomous level of the operation and maintenance network, so that the transparency, fairness and reliability of the hierarchical evaluation can be enhanced.

Example two

Fig. 6 illustrates an operation and maintenance network architecture, and further illustrates that, based on the operation and maintenance network illustrated in fig. 1, a third party evaluation system is included in the NMS of the operation and maintenance network, where the third party evaluation system has a network management function and can be used to evaluate the network autonomous level of the operation and maintenance link. The EMS in the operation and maintenance network includes some functional devices with network element management functions, including, for example, an open management function (exposure governance management function, EGMF) device, an IDMF producer device, and MnF devices interacting with the EGMF device, the IDMF producer device, for example, the MnF devices include DCCF devices, MDAF devices, and the like.

The EGMF device, the IDMF producer device, the DCCF device, and the MDAF device may be referred to as EGMF, IDMF producer, DCCF, and MDAF, respectively. Wherein. The functionality of IDMF producer, DCCF and MDAF in network coverage optimization may be understood with reference to the description of example one. The EGMF may be responsible for opening MnF data or services to third party uses, such as sending MnF assessment data to a third party assessment system.

Corresponding to the communication method described in fig. 3, EGMF may be understood as an example of the aforementioned first device, and IDMF producer, DCCF, MDAF are examples of the aforementioned second device, i.e., the second device includes IDMF producer, DCCF, and MDAF. The third party evaluation system may be understood as an example of the aforementioned third device, or the devices in the third party evaluation system may be understood as an example of the third device.

In the second example, the EGMF obtains evaluation data of N operation and maintenance tasks in the network coverage optimization from the IDMF producer, the DCCF and the MDAF, and sends the obtained evaluation data of N operation and maintenance tasks to the third party evaluation system, so that the third party evaluation system determines the network autonomous level of the network coverage optimization according to the evaluation data of N operation and maintenance tasks in the network coverage optimization. Optionally, the value of N is the number of all operation and maintenance tasks in the network coverage optimization.

Alternatively, EGMF may be understood as a device in EMS. Referring to the scheme described in case two in S301, the EGMF may specifically transmit the first information to one device of the IDMF producer, DCCF, and MDAF; the device which receives the first information sends the evaluation data of the operation and maintenance tasks of the corresponding part of the signaling request to the rest of the execution entities; and further, the IDMF producer, the DCCF, and the MDAF send signaling to the EGMF to indicate the evaluation data of the operation and maintenance tasks of the respective corresponding portions, and the EGMF may obtain the third information according to the signaling sent from the IDMF producer, the DCCF, and the MDAF. The third information is used to indicate evaluation data of the N operation and maintenance tasks in the network coverage optimization. Further, the EGMF sends evaluation data for the N operation and maintenance tasks in the network coverage optimization to the third party evaluation system.

As an example, fig. 7 illustrates a method for evaluating network coverage optimization, which is an example of the method illustrated in fig. 3, and may be applied to the operation and maintenance network illustrated in fig. 6. The method mainly comprises the following steps:

s701, the third party evaluation system sends second information to the EGMF.

Wherein this step corresponds to S300 in fig. 3, the second information is used to request an evaluation of a network autonomous level of network coverage optimization.

Alternatively, the second information may be substituted for a data open request described as a network coverage optimization hierarchy assessment. The second information may include first information, where the first information is used to request evaluation data of N operation and maintenance tasks for network coverage optimization.

For example, the coverage optimization evaluation indication may contain information indicating network coverage optimized operation and maintenance tasks, such as a Task set list, to represent evaluation data that requires each operation and maintenance Task in the Task set list. The Task set list includes the identification of N operation and maintenance tasks in the network coverage optimization: intent translation, control information generation, intent assessment, data collection, problem identification (or anomaly identification), degradation prediction, problem delimitation, root cause analysis, optimization scheme generation, optimization scheme assessment and decision, and optimization scheme delivery. The coverage optimization evaluation indication may also contain an identification of the Task execution mode (or Task man-machine division status) to indicate the execution mode of the operation and maintenance Task included in the evaluation data required for each operation and maintenance Task. Optionally, the coverage optimization evaluation indication may further include first indication information, that is, a candidate mode of the Task execution mode.

S702, the EGMF sends network coverage optimization creation intention and first information to an IDMF producer.

Specifically, the network coverage optimization hierarchical assessment intention may be understood with reference to the description of S501, the first information may alternatively be described as the network coverage optimization creation intention, and the definition of the first information may be understood with reference to the description of S701, which will not be described in detail in the embodiments of the present application.

S703, the IDMF producer translates the intention according to the network coverage optimization creation intention and the first information, generates a related strategy of the network coverage optimization, and establishes a corresponding relation between the operation and maintenance task of the network coverage optimization and the execution entity.

Specifically, this step may be performed with reference to S503, which is not described in detail in the embodiment of the present application.

S704, IDMF producer sends evaluation data intended for translation to EGMF.

Specifically, the evaluation data of the intent translation may be understood with reference to the description of S504, and this will not be described in detail in the embodiments of the present application.

Further, corresponding to the second case described in S301, the IDMF producer may be understood as an example of the fourth device, and may request the evaluation data of the part of the operation and maintenance task processed by the IDMF producer from the rest of the execution entities (i.e., devices in the EMS) except for itself. Specifically, it can be understood with reference to the following steps S705 to S720:

S705, the IDMF producer sends a data collection request message to the DCCF.

Specifically, the data collection request message may be understood with reference to the description of S505, which is not described in detail in the embodiment of the present application.

S706, the DCCF collects data from the relevant network elements according to the data collection request message.

S707, the DCCF feeds back the collected data to the IDMF producer. For example, DCCF feeds back collected performance data, configuration data, etc. to IDMF producer.

In a possible case where the data collected by the DCCF from the relevant network element is part of the data requested by the IDMF producer, the DCCF may also request the data that cannot be received from the relevant network element from the outside. Optionally, the method further includes S708, S708 as an optional step after performing S707, which is illustrated in fig. 7 by a dashed line.

S708, the IDMF producer receives an externally imported data list input by an operator (operator), the externally imported data list including stock data, environment data, and the like. S708 is illustrated in dashed lines in fig. 7 as an optional step. Alternatively, the request for the externally imported data list may be triggered by the DCCF.

S709, the DCCF transmits the evaluation data of the data collection to the EGMF.

The evaluation data of the data collection may be understood with reference to the description of S510, which is not described in detail in the embodiment of the present application.

S710, the IDMF producer sends a network coverage optimization request message to the MDAF.

Specifically, this step may be performed with reference to S511, which is not described in detail in the embodiment of the present application.

S711, the MDAF processes a plurality of operation and maintenance tasks of network coverage optimization according to the network coverage optimization request message.

Specifically, the MDAF may accomplish multiple operation and maintenance tasks, such as problem identification, problem delimitation, degradation prediction, root cause analysis, and solution generation, independently or with the aid of EGMF.

Further, the MDAF may send evaluation data of the plurality of operation and maintenance tasks processed by the MDAF to the EGMF, including steps S712 and S713, that is, the evaluation data of the plurality of operation and maintenance tasks processed by the MDAF includes execution modes and execution effects of the plurality of operation and maintenance tasks. It is understood that the evaluation data of the plurality of operation and maintenance tasks includes the evaluation data of the operation and maintenance tasks processed by the MDAF among the N operation and maintenance tasks of the network coverage optimization indicated by the third information.

S712, the MDAF sends the EGMF an execution mode of the plurality of operation and maintenance tasks.

Similarly, taking the problem identification in multiple operation and maintenance tasks as an example, the evaluation data of the problem identification may include a notification of completion of the problem identification and Task labor division status (e.g., system) of the problem identification.

S713, the MDAF sends the EGMF with the execution effect of the plurality of operation and maintenance tasks.

Specifically, this step may be performed with reference to S514, which is not described in detail in the embodiment of the present application.

S714, the IDMF producer performs optimization scheme evaluation and decision.

Specifically, the IDMF producer can complete the evaluation and decision of the optimization scheme independently or with the aid of EGMF, such as analyzing the coverage optimization effect of the optimization scheme, conflicts with other intents, and the like.

S715, the IDMF producer sends evaluation data of the optimization scheme evaluation and decision to the EGMF.

The evaluation data of the optimization scheme evaluation and decision may be understood with reference to the description in S516, and this will not be described in detail in the embodiments of the present application.

And S716, the IDMF producer transmits the optimization scheme to the relevant network element. Specifically, the IDMF producer may indicate a network element list corresponding to the optimization scheme and an operation list of the optimization scheme to the relevant network elements.

S717, the IDMF producer sends evaluation data issued by the optimization scheme to the EGMF.

The evaluation data issued by the optimization scheme can be understood with reference to the description in S518, and this will not be described in detail in the embodiment of the present application.

S718, the IDMF producer performs intention evaluation.

Specifically, this step may be performed with reference to S519, which is not described in detail in the embodiments of the present application.

S719, IDMF producer sends evaluation data for the intention evaluation to EGMF.

The evaluation data of the intention evaluation may be understood with reference to the description in S520, and this will not be described in detail in the embodiments of the present application.

S720, the IDMF producer feeds back the result of the intention evaluation to the EGMF. Wherein the result of the intent assessment may be used to indicate the specifics of the intent to complete. Alternatively, the result of the intention evaluation may also be used as evaluation data for evaluating the network autonomy level of the network coverage optimization.

It is understood that the content indicated by the third information in the present example two includes the evaluation data referred to in S704, S709, S712, S715, S717, S719, and S720, corresponding to the second device described in S301 transmitting the third information to the first device.

And S721, the EGMF sends the received evaluation data of the N operation and maintenance tasks in the network coverage optimization to a third party evaluation system.

S722, the third party evaluation system determines the network autonomy level of the operation link of the network coverage optimization according to the received evaluation data of each operation task in the network coverage optimization.

Specifically, the step S722 may be implemented with reference to the determination scheme of the network autonomous level in the first operation and maintenance link described in S302A, which is not described in detail in the embodiment of the present application.

In the second example, the third party evaluation system is supported to collect the evaluation data of each operation and maintenance task through the EGMF, and then the third party evaluation system completes the evaluation of the network autonomous level of the operation and maintenance network, so that the evaluation data is not required to be exposed to the IDMF consumer, the data security can be protected, and the privacy leakage problem is avoided.

Example three

Fig. 8 illustrates an operation and maintenance network architecture, and further illustrates that the NMS includes an IDMF consumer, and the EMS includes an IDMF producer, DCCF, and MDAF, based on the operation and maintenance network illustrated in fig. 1. The definitions of IDMF consumer, IDMF producer, DCCF, and MDAF can be understood with reference to the description of fig. 4, which will not be described in detail in the embodiments of the present application. Optionally, the operation and maintenance network illustrated in fig. 8 may also include EVA. The difference from fig. 4 is that the EVA in fig. 8 is deployed in the IDMF producer, and the IDMF producer or the EVA in the IDMF producer may be responsible for summarizing the evaluation data of each operation and maintenance task in the network coverage optimization, to determine whether a certain operation and maintenance task is executed or not and the corresponding execution mode.

In example three, the foregoing first apparatus may correspond to the IDMF producer or EVA in the IDMF producer. In addition, IDMF, DCCF, and MDAF are examples of the aforementioned second device, or may be described as the second device including IDMF, DCCF, MDAF, and the like. It is to be understood that example three corresponds to the case where the first device is included in the second device as described in S301. Further, IDMF consumer is an example of the third apparatus.

In this example three, corresponding to the description in S301, the IDMF consumer may send second information to the IDMF producer requesting to evaluate the network autonomy level of network coverage optimization. The IDMF producer acquires the evaluation data of the N operation and maintenance tasks in network coverage optimization from the IDMF producer, the DCCF and the MDAF, and then the IDMF producer sends the acquired evaluation data of the N operation and maintenance tasks to the IDMF consumer. The IDMF consumer can determine a network autonomous level of the network coverage optimization according to the evaluation data of the N operation and maintenance tasks in the network coverage optimization. Optionally, the value of N is the number of all operation and maintenance tasks in the network coverage optimization.

Specifically, the IDMF producer itself may process a portion of the N operation and maintenance tasks. The IDMF producer may send signaling requests to DCCF and MDAF evaluation data of the DCCF and the MDAF processed partial operation and maintenance tasks. Based on this, the IDMF producer, DCCF, and MDAF send signaling to the EGMF to indicate the evaluation data of the respective corresponding portion of the operation and maintenance tasks, and the IDMF consumer may obtain third information according to the signaling sent from the IDMF producer, DCCF, and MDAF, where the third information is used to indicate the evaluation data of the N operation and maintenance tasks in the network coverage optimization.

As an example, fig. 9 illustrates a method for evaluating network coverage optimization, which is an example of the method illustrated in fig. 3, and may be applied to the operation and maintenance network illustrated in fig. 8. The method mainly comprises the following steps:

s901, the IDMF consumer sends second information to the IDMF producer.

Wherein this step corresponds to S300 in fig. 3, the second information is used to request an evaluation of a network autonomous level of network coverage optimization. Alternatively, the second information may alternatively be described as network coverage optimization rating intent.

Optionally, the second information may include first information, where the first information is used to request evaluation data of N operation and maintenance tasks for network coverage optimization. The first information may include a coverage optimization evaluation indication, where the coverage optimization evaluation indication is used to indicate evaluation data required for evaluating the operation and maintenance link of network coverage optimization; alternatively, coverage optimization evaluation may be understood as evaluation data indicating N operation and maintenance tasks for indicating the first information request.

Alternatively, when the IDMF producer is configured with EVA, the IDMF consumer sends the second information to the EVA in the IDMF producer.

S902, the IDMF consumer sends a network coverage optimization creation intention to the IDMF producer.

The definition of the network coverage optimization creation intention may be understood with reference to the description in S501, and this will not be described in detail in the embodiment of the present application.

Alternatively, S901 and S902 may be performed simultaneously, or may be performed in a time-sharing manner, which is not limited in the embodiment of the present application.

S903, the IDMF producer translates the intention of creating the network coverage optimization and the second information, generates a related strategy of the network coverage optimization, and establishes a corresponding relation between the operation and maintenance task of the network coverage optimization and the execution entity.

S904, IDMF producer determines evaluation data intended for translation.

Specifically, this step may be performed with reference to S504, which is not described in detail in the embodiment of the present application.

Further, the IDMF producer may request the evaluation data of the part of the operation and maintenance tasks processed by the IDMF producer from the rest of the execution entities (i.e., devices in the EMS) except for itself according to the first information in the second information. Specifically, it can be understood with reference to the following steps:

S905, the IDMF producer sends a data collection request message to the DCCF.

Specifically, this step may be performed with reference to S505, which is not described in detail in the embodiments of the present application.

S906, the DCCF collects data from the relevant network elements according to the data collection request message.

S907, DCCF feeds back the collected data to IDMF producer. For example, DCCF feeds back collected performance data, configuration data, etc. to IDMF producer.

In a possible case where the data collected by the DCCF from the relevant network element is part of the data requested by the IDMF producer, the DCCF may also request the data that cannot be received from the relevant network element from the outside. Optionally, the above method further comprises S908 to S909, S908 to S909 as optional steps after execution of S907, which are illustrated in fig. 9 by dashed lines.

S908, the DCCF transmits an external data import request message to the IDMF consumer, which may include a data list, such as stock data, environment data, etc., requesting external import.

S909, the IDMF consumer sends the externally imported data list to the IDMF producer.

S910, the DCCF sends the evaluation data of the data collection to the IDMF producer.

Specifically, the evaluation data of the data collection may be understood with reference to S510, which is not described in detail in the embodiments of the present application.

Optionally, when EVA is configured in the IDMF producer, the DCCF specifically sends evaluation data of the data collection to the EVA.

S911, the IDMF producer sends a network coverage optimization request message to the MDAF.

S912, the MDAF processes a plurality of operation and maintenance tasks of network coverage optimization according to the network coverage optimization request message.

Specifically, this step may be performed with reference to S512, which is not described in detail in the embodiment of the present application.

Further, the MDAF may send evaluation data of the plurality of operation and maintenance tasks processed by the MDAF producer, including steps S913 and S914, that is, the evaluation data of the plurality of operation and maintenance tasks processed by the MDAF includes execution modes and execution effects of the plurality of operation and maintenance tasks. It is understood that the evaluation data of the plurality of operation and maintenance tasks includes the evaluation data of the operation and maintenance tasks processed by the MDAF among the N operation and maintenance tasks of the network coverage optimization indicated by the third information.

S913, the MDAF sends the IDMF producer an execution mode of the plurality of operation and maintenance tasks.

Similarly, taking the problem identification in multiple operation and maintenance tasks as an example, the evaluation data of the problem identification may include a notification of completion of the problem identification and Task labor division status (e.g., system) of the problem identification. Optionally, when EVA is configured in the IDMF producer, the MDAF specifically sends the EVA an execution mode of the plurality of operation and maintenance tasks.

S914, the MDAF sends the execution effect of the operation and maintenance tasks to the IDMF producer. Examples of some of the effects of the execution are provided below: such as problem identification results (weak coverage), problem delimitation (cells where weak coverage occurs), root cause analysis results (power setting is unreasonable), scheme generation results (providing antenna power to set values). As an optional step, S914 is illustrated with a broken line in fig. 10.

S915, the IDMF producer performs optimization scheme evaluation and decision.

Specifically, this step may be performed with reference to S515, which is not described in detail in the embodiments of the present application.

S916, the IDMF producer determines evaluation data for the optimization scheme evaluation and decision.

Specifically, this step may be performed with reference to S516, which is not described in detail in the embodiment of the present application.

S917, the IDMF producer issues the optimization scheme to the relevant network elements.

Specifically, this step may be performed with reference to S517, which is not described in detail in the embodiments of the present application.

S918, the IDMF producer determines evaluation data issued by the optimization scheme.

Specifically, the evaluation data issued by the optimization scheme may be understood with reference to S518, which is not described in detail in the embodiment of the present application.

S919, IDMF producer performs intent evaluation.

S920, IDMF producer determines evaluation data for the intended evaluation.

Specifically, this step may be performed with reference to S520, which is not described in detail in the embodiments of the present application.

S921, the IDMF producer feeds back the result of the intention evaluation to the IDMF consumer. Wherein the result of the intent assessment may be used to indicate the specifics of the intent to complete.

S922, the IDMF producer determines the network autonomy level of the operation link of the network coverage optimization according to the evaluation data of each operation task in the network coverage optimization.

Specifically, the step S922 may be implemented with reference to the determination scheme of the network autonomous level in the first operation and maintenance link described in S302A, which is not described in detail in the embodiment of the present application.

S923, the IDMF producer sends the network autonomous level of network coverage optimization to the IDMF consumer.

In the second example, the IDMF producer is supported to collect the evaluation data of each operation and maintenance task, and complete the evaluation of the network autonomous level of the operation and maintenance network, so that the evaluation data is not required to be exposed to the IDMF consumer, the data security can be protected, and the privacy leakage problem can be avoided. And meanwhile, operators or equipment providers corresponding to the operation and maintenance network can conveniently evaluate the operation and maintenance network internally, and the improvement and improvement of the automation capacity of the operation and maintenance network are assisted.

Based on the same concept, referring to fig. 10, an embodiment of the present application provides an evaluation apparatus 1000 of an operation and maintenance network autonomous level, where the evaluation apparatus 1000 of an operation and maintenance network autonomous level includes a processing module 1001 and a communication module 1002. The evaluation device 1000 of the autonomous level of the operation and maintenance network may be a first device, or may be an evaluation device of the autonomous level of the operation and maintenance network applied to or matched with the first device, which can implement a communication method executed by the first device; alternatively, the evaluation device 1000 of the autonomous level of the operation and maintenance network may be a second device, or may be an evaluation device of the autonomous level of the operation and maintenance network applied to or matched with the second device, which can implement a communication method executed by the second device; alternatively, the evaluation device 1000 of the autonomous level of the operation and maintenance network may be a third device, or may be an evaluation device of the autonomous level of the operation and maintenance network applied to or matched with the third device, which can implement a communication method executed by the third device; alternatively, the evaluation device 1000 of the autonomous level of the operation and maintenance network may be a fourth device, or may be an evaluation device of the autonomous level of the operation and maintenance network applied to or used in cooperation with the fourth device, which is capable of implementing a communication method executed on the fourth device side.

The communication module may also be referred to as a transceiver module, a transceiver, or a transceiver device. A processing module may also be called a processor, a processing board, a processing module, a processing device, or the like. Optionally, the communication module is configured to perform the sending operation and the receiving operation on the first device side or the second device side in the above method, where a device for implementing a receiving function in the communication module may be regarded as a receiving unit, and a device for implementing a sending function in the communication module may be regarded as a sending unit, that is, the communication module includes the receiving unit and the sending unit.

Taking the first device as an example, when the evaluation device 1000 of the autonomous level of the operation and maintenance network is applied to the first device, the processing module 1001 may be configured to implement the processing function of the first device in the example shown in fig. 3, and the communication module 1002 may be configured to implement the transceiving function of the first device in the example shown in fig. 3. Or the evaluation device of the autonomous level of the operation and maintenance network can be understood by referring to the third aspect of the invention and possible designs in the third aspect.

Taking the second device as an example, when the evaluation device 1000 of the autonomous level of the operation and maintenance network is applied to the second device, the processing module 1001 may be configured to implement the processing function of the second device in the example shown in fig. 3, and the communication module 1002 may be configured to implement the transceiving function of the second device in the example shown in fig. 3. Or the evaluation device of the autonomous level of the operation and maintenance network can be understood by referring to the fourth aspect of the invention and the possible designs in the fourth aspect.

Furthermore, it should be noted that the foregoing communication module and/or the processing module may be implemented by a virtual module, for example, the processing module may be implemented by a software functional unit or a virtual device, and the communication module may be implemented by a software functional unit or a virtual device. Alternatively, the processing module or the communication module may be implemented by an entity device, for example, if the device is implemented by a chip/chip circuit, the communication module may be an input/output circuit and/or a communication interface, and perform an input operation (corresponding to the foregoing receiving operation) and an output operation (corresponding to the foregoing sending operation); the processing module is an integrated processor or microprocessor or integrated circuit.

The division of the modules in the embodiments of the present application is schematically shown, which is merely a logic function division, and may have another division manner when actually implemented, and in addition, each functional module in each example of the embodiments of the present application may be integrated in one processor, or may exist separately and physically, or may be integrated in one module by two or more modules. The integrated modules may be implemented in hardware or in software functional modules.

Based on the same technical concept, the embodiment of the application also provides an evaluation device 1100 for the autonomous level of the operation and maintenance network. For example, the evaluation device 1100 of the autonomous level of the operation and maintenance network may be a chip or a chip system. Alternatively, the chip system in the embodiments of the present application may be formed by a chip, and may also include a chip and other discrete devices.

The evaluation device 1100 of the autonomous level of the operation and maintenance network may be used to implement the functionality of any network element in the communication system described in the previous example. The evaluation device 1100 for the level of autonomy of the operation and maintenance network may comprise at least one processor 1110, the processor 1110 being coupled to a memory, the memory may alternatively be located within the device, the memory may be integrated with the processor, and the memory may also be located outside the device. For example, the evaluation device 1100 for an autonomous level of an operation and maintenance network may further include at least one memory 1120. Memory 1120 holds the computer programs, computer programs or instructions and/or data necessary to implement any of the examples described above; processor 1110 may execute a computer program stored in memory 1120 to perform the methods of any of the examples described above.

The evaluation device 1100 of the autonomous level of the operation and maintenance network may further include a communication interface 1130, and the evaluation device 1100 of the autonomous level of the operation and maintenance network may perform information interaction with other devices through the communication interface 1130. By way of example, the communication interface 1130 may be a transceiver, circuit, bus, module, pin, or other type of communication interface. When the evaluation device 1100 of the operation and maintenance network autonomous level is a chip-type device or circuit, the communication interface 1130 in the evaluation device 1100 of the operation and maintenance network autonomous level may be an input/output circuit, and may input information (or called receiving information) and output information (or called transmitting information), and the processor may be an integrated processor or a microprocessor or an integrated circuit or a logic circuit, and the processor may determine the output information according to the input information.

The coupling in the embodiments of the present application is an indirect coupling or communication connection between devices, units, or modules, which may be in electrical, mechanical, or other forms for information interaction between the devices, units, or modules. Processor 1110 may operate in conjunction with memory 1120 and communication interface 1130. The specific connection medium between the processor 1110, the memory 1120, and the communication interface 1130 is not limited in the embodiments of the present application.

Optionally, referring to fig. 11, the processor 1110, the memory 1120, and the communication interface 1130 are connected to each other through a bus 1140. The bus 1140 may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is shown in FIG. 11, but not only one bus or one type of bus.

In the embodiments of the present application, the processor may be a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic blocks disclosed in the embodiments of the present application. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in the processor for execution.

In the embodiment of the present application, the memory may be a nonvolatile memory, such as a hard disk (HDD) or a Solid State Drive (SSD), or may be a volatile memory (volatile memory), for example, a random-access memory (RAM). The memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory in the embodiments of the present application may also be circuitry or any other device capable of implementing a memory function for storing program instructions and/or data.

In a first possible embodiment, the evaluation device 1100 of the autonomous level of the operation and maintenance network may be applied to a first device, and the evaluation device 1100 of the autonomous level of the specific operation and maintenance network may be the first device or may be a device capable of supporting the first device, and implementing the function of the first device in any of the above-mentioned examples. Memory 1120 holds computer programs (or instructions) and/or data that implement the functions of the first device in any of the examples described above. Processor 1110 may execute a computer program stored by memory 1120 to perform the method performed by the first apparatus in any of the examples described above. The communication interface in the evaluation device 1100, applied to the first device, of the autonomous level of the operation and maintenance network may be used to interact with the second device, send information to the second device, or receive information from the second device.

In a second possible embodiment, the evaluation device 1100 of the autonomous level of the operation and maintenance network may be applied to a second device, and the evaluation device 1100 of the autonomous level of the operation and maintenance network may be a second device or a device capable of supporting the second device, and implementing the function of the second device in any of the above-mentioned examples. Memory 1120 holds computer programs (or instructions) and/or data that implement the functions of the second device in any of the examples described above. Processor 1110 may execute a computer program stored by memory 1120 to perform the method performed by the second apparatus in any of the examples described above. Applied to a second device, the communication interface in the evaluation device 1100 of the autonomous level of the operation and maintenance network can be used to interact with the first device, send information to the first device or receive information from the first device.

In a third possible embodiment, the evaluation device 1100 of the autonomous level of the operation and maintenance network may be applied to a third device, and the evaluation device 1100 of the autonomous level of the operation and maintenance network may be the third device, or may be a device capable of supporting the third device, and implementing the function of the third device in any of the above-mentioned examples. Memory 1120 holds computer programs (or instructions) and/or data that implement the functions of the third apparatus in any of the examples described above. Processor 1110 may execute a computer program stored by memory 1120 to perform the method performed by the third apparatus in any of the examples described above. Applied to a second device, the communication interface in the evaluation device 1100 of the autonomous level of the operation and maintenance network can be used to interact with the first device, send information to the first device or receive information from the first device.

Since the evaluation device 1100 for an autonomous level of an operation and maintenance network provided in this example may be applied to a first device, a method performed by the first device is completed, or applied to a second device, a method performed by the second device is completed, or applied to a third device, a method performed by the third device is completed. Therefore, reference may be made to the above method examples for the technical effects that can be obtained, and they will not be described herein.

Based on the above examples, the embodiments of the present application provide a communication system including a first apparatus, a second apparatus, and a third apparatus. Wherein the first apparatus, the second apparatus, and the third apparatus may implement the method provided in the example shown in fig. 3.

The technical solution provided in the embodiments of the present application may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, a terminal device, an access network device, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (digital video disc, DVD)), or a semiconductor medium, etc.

In the embodiments of the present application, the examples may refer to each other without logical contradiction, for example, methods and/or terms between method embodiments may refer to each other, for example, functions and/or terms between apparatus examples and method examples may refer to each other.

It will be apparent to those skilled in the art that various modifications and variations can be made to the embodiments of the present application without departing from the scope of the embodiments of the present application. Thus, given that such modifications and variations of the embodiments of the present application fall within the scope of the claims of the embodiments of the present application and their equivalents, the embodiments of the present application are intended to include such modifications and variations as well.

Claims

1. An evaluation method for an autonomous level of an operation and maintenance network, comprising the steps of:

the first device acquires evaluation data of N operation and maintenance tasks in a first operation and maintenance operation link from the second device, wherein N is a positive integer;

the first device sends evaluation data of the N operation and maintenance tasks to a third device, wherein the evaluation data of the N operation and maintenance tasks are used for determining the network autonomy level of the first operation and maintenance operation link; or,

And the first device determines the network autonomy level of the first operation link according to the evaluation data of the N operation tasks.

2. The method as recited in claim 1, further comprising:

the first device sends first information to the second device, wherein the first information is used for requesting evaluation data of N operation and maintenance tasks in a first operation and maintenance operation link, and N is a positive integer.

3. The method as recited in claim 2, further comprising:

the first device receives second information from the third device, the second information being used for requesting evaluation of a network autonomy level of the first operation and maintenance link, the second information including the first information.

4. A method according to claim 2 or 3, wherein the evaluation data of any one of the N operation and maintenance tasks of the first information request comprises one or more of the following: the information of any operation and maintenance task, the execution mode of any operation and maintenance task, and the execution effect of any operation and maintenance task.

5. The method of any one of claims 2-4, wherein the first information includes first indication information, where the first indication information is used to indicate a candidate mode corresponding to an execution mode of any one of the N operation and maintenance tasks, and the candidate mode includes: a manual execution mode, a manual and system common execution mode, and a system execution mode.

6. The method according to any one of claims 2 to 5, wherein the first information includes index information for measuring an execution effect of some or all of the N operation and maintenance tasks.

7. The method of any of claims 1-6, wherein the first device obtaining, from the second device, evaluation data for N operation and maintenance tasks in the first operation and maintenance operation link, comprising:

the first device receives third information from the second device, the third information indicating one or more of: and the processing of the N operation and maintenance tasks is completed, the execution mode of the N operation and maintenance tasks is completed, and the execution effect of part or all of the N operation and maintenance tasks is achieved.

8. An evaluation method for an autonomous level of an operation and maintenance network, comprising the steps of:

the second device receives first information from the first device, wherein the first information is used for requesting evaluation data of N operation and maintenance tasks in a first operation and maintenance operation link, and N is a positive integer;

and the second device sends third information to the first device according to the first information, wherein the third information is used for indicating the evaluation data of the N operation and maintenance tasks.

9. The method of claim 8, wherein the evaluation data of any one of the N operation and maintenance tasks of the first information request includes one or more of: the information of any operation and maintenance task, the execution mode of any operation and maintenance task, and the execution effect of any operation and maintenance task.

10. The method of claim 9, wherein the first information includes first indication information, and the first indication information is used for indicating a candidate mode corresponding to the execution mode of the any operation and maintenance task, where the candidate mode includes: a manual execution mode, a manual and system common execution mode, and a system execution mode.

11. A method according to claim 9 or 10, wherein the first information includes index information for measuring the execution effect of some or all of the N operation and maintenance tasks.

12. The method of any of claims 8-11, wherein the third information is to indicate one or more of: and the processing of the N operation and maintenance tasks is completed, the execution mode of the N operation and maintenance tasks is completed, and the execution effect of part or all of the N operation and maintenance tasks is achieved.

13. An evaluation method for an autonomous level of an operation and maintenance network, comprising the steps of:

the second device sends evaluation data of N operation and maintenance tasks in the first operation and maintenance operation link to the first device, wherein N is a positive integer;

the first device determines the network autonomy level of the first operation link according to the evaluation data of the N operation tasks; or,

the first device sends the evaluation data of the N operation and maintenance tasks to a third device;

and the third device determines the network autonomy level of the first operation link according to the evaluation data of the N operation tasks.

14. The method as recited in claim 13, further comprising:

the first device sends first information to the second device, wherein the first information is used for requesting evaluation data of N operation and maintenance tasks in a first operation and maintenance operation link.

15. The method as recited in claim 14, further comprising:

the third device sends second information to the first device, the second information is used for requesting to evaluate the network autonomous level of the first operation link, and the second information comprises the first information.

16. The method of any of claims 13-15, wherein the second device sending, to the first device, evaluation data for N operation and maintenance tasks in the first operation and maintenance operation link, comprising:

The second device sends third information to the first device, the third information being used to indicate one or more of: and the processing of the N operation and maintenance tasks is completed, the execution mode of the N operation and maintenance tasks is completed, and the execution effect of part or all of the N operation and maintenance tasks is achieved.

17. An operation and maintenance network autonomous level assessment device, comprising:

the processing module is used for acquiring evaluation data of N operation and maintenance tasks in the first operation and maintenance operation link from the second device, wherein N is a positive integer;

18. An operation and maintenance network autonomous level assessment device, comprising:

the communication module is used for receiving first information from the first device, wherein the first information is used for requesting evaluation data of N operation and maintenance tasks in a first operation and maintenance operation link, and N is a positive integer;

19. An operation and maintenance network autonomous level assessment device, comprising:

a processor coupled to a memory for invoking computer program instructions stored in the memory to perform the method of any of claims 1-7, or the method of any of claims 8-12, or the method of any of claims 13-16.

20. An operation and maintenance network autonomous level evaluation system, characterized by comprising the operation and maintenance network autonomous level evaluation device according to claim 17, and the operation and maintenance network autonomous level evaluation device according to claim 18.

21. A computer readable storage medium having instructions stored thereon which, when run on a computer, cause the computer to perform the method of any one of claims 1-7 or the method of any one of claims 8-12 or the method of any one of claims 13-16.

22. A computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 7, or the method of any one of claims 8 to 12, or the method of any one of claims 13 to 16.