FI129282B

FI129282B - Automated monitoring and control of communication networks

Info

Publication number: FI129282B
Application number: FI20205508A
Authority: FI
Inventors: Jukka-Pekka Salmenkaita; Petteri Lundèn
Original assignee: Elisa Oyj
Priority date: 2020-05-20
Filing date: 2020-05-20
Publication date: 2021-11-15
Also published as: WO2021234216A1; FI20205508A1

Abstract

A computer implemented method of monitoring and controlling a communication network. Information about results of experts’ evaluation of machine generated suggestions is collected (304, 309), wherein the machine generated suggestions relate to making changes in the communication network; building and updating expert profiles based on the collected information; and using the expert profiles for processing further machine generated suggestions.

Description

AUTOMATED MONITORING AND CONTROL OF COMMUNICATION NETWORKS

TECHNICAL FIELD The present disclosure generally relates to automated monitoring and control of communication networks. The disclosure relates particularly, though not exclusively, to automated monitoring and control that involves interaction with human experts.

BACKGROUND — This section illustrates useful background information without admission of any technique described herein representative of the state of the art. Cellular communication networks are complex systems comprising a plurality of cells serving users of the network. When users of the communication network move in the area of the network, connections of the users are seamlessly handed over between cells of the network. There are various factors that affect operation of individual cells and co-operation between the cells. In order for the communication network to operate as intended and to provide planned quality of service, cells of the communication network need to operate as planned. For example, the cells need to provide sufficient coverage without too much interfering with operation of neighboring cells. In general, operation of the communication networks is continuously monitored and N controlled in order to detect any problems in operation of the network as soon as dS possible so that operation of the network can be optimized. Various automated = methods have been developed for this purpose to improve efficiency and accuracy. N 25 Many automated solutions still reguire human experts to evaluate and control E automated actions. There is constant desire to further develop automated 2 monitoring and control methods.

S S

SUMMARY The appended claims define the scope of protection. Any examples and technical descriptions of apparatuses, products and/or methods in the description and/or drawings not covered by the claims are presented not as embodiments of the invention but as background art or examples useful for understanding the invention. According to a first example aspect there is provided a computer implemented method of monitoring and controlling a communication network. The method comprises collecting information about results of experts’ evaluation of machine generated suggestions, wherein the machine generated suggestions relate to making changes in the communication network; building and updating expert profiles based on the collected information; and using the expert profiles for processing further machine generated suggestions. In an example embodiment, the expert profile of an expert comprises an identifier of the expert and information about successfulness of evaluations of the expert. In an example embodiment, the expert profile further comprises information about successfulness of evaluations of the expert for different case types. In an example embodiment, the information about successfulness of evaluations of the expert comprises information about whether a change approved by the expert is maintained or reverted. S In an example embodiment, the information about successfulness of evaluations of

O N the expert comprises information about whether the expert's evaluations differ from = evaluations of other experts.

N - 25 In an example embodiment, the expert profile of an expert comprises information a > about reliability of the expert profile. 00

O 2 In an example embodiment, the method further comprises

O

QA S storing a plurality of expert profiles to form an expert database;

rating the expert profiles in relation to other expert profiles in the expert database based on information about successfulness of evaluations of the experts and/or based on reliability of the expert profiles; and using the ratings of the expert profiles when using the expert profiles for processing further machine generated suggestions. In an example embodiment, the machine generated suggestions specify concrete changes to be performed in the communication network. In an example embodiment, the machine generated suggestions specify a part of network where changes are needed. The machine generated suggestions may specify for example a cell, a base station or an area where problems are detected. In an example embodiment, using the expert profiles comprises weighting evaluation results of experts based on the respective expert profiles. In an example embodiment, the method further comprises providing feedback to generation of the machine generated suggestions based on the results of the experts’ evaluation to adjust the generation of the further machine generated suggestions, wherein using the expert profiles comprises weighting the feedback based on the respective expert profiles. In an example embodiment, using the expert profiles comprises choosing an expert for evaluating a further machine generated suggestion based on the expert profiles. In an example embodiment, using the expert profiles comprises choosing an additional expert for evaluating a further machine generated suggestion based on o the expert profiles.

QA N According to a second example aspect of the present invention, there is provided 3 an apparatus comprising a processor and a memory including computer program code; the memory and the computer program code configured to, with the E processor, cause the apparatus to perform the method of the first aspect or any 00 related embodiment.

LO S According to a third example aspect of the present invention, there is provided a computer program comprising computer executable program code which when executed by a processor causes an apparatus to perform the method of the first aspect or any related embodiment. According to a fourth example aspect there is provided a computer program product comprising a non-transitory computer readable medium having the computer program of the third example aspect stored thereon. According to a fifth example aspect there is provided an apparatus comprising means for performing the method of the first aspect or any related embodiment. Any foregoing memory medium may comprise a digital data storage such as a data disc or diskette, optical storage, magnetic storage, holographic storage, opto- magnetic storage, phase-change memory, resistive random access memory, magnetic random access memory, solid-electrolyte memory, ferroelectric random access memory, organic memory or polymer memory. The memory medium may be formed into a device without other substantial functions than storing memory or it may be formed as part of a device with other functions, including but not limited to a memory of a computer, a chip set, and a sub assembly of an electronic device. Different non-binding example aspects and embodiments have been illustrated in the foregoing. The embodiments in the foregoing are used merely to explain selected aspects or steps that may be utilized in different implementations. Some embodiments may be presented only with reference to certain example aspects. It should be appreciated that corresponding embodiments may apply to other example aspects as well.

BRIEF DESCRIPTION OF THE FIGURES N Some example embodiments will be described with reference to the accompanying N figures, in which: = 25 Fig. 1 schematically shows an example scenario according to an example - embodiment; E Fig. 2 shows a block diagram of an apparatus according to an example 2 embodiment; and S Figs. 3-5 show flow diagrams illustrating example methods according to certain N 30 embodiments.

DETAILED DESCRIPTION In the following description, like reference signs denote like elements or steps.

Automated monitoring and control methods may be based on artificial intelligence 5 and machine learning technologies. The methods are often based on iterations of machine generated suggestions of changes or actions or places where changes are potentially needed and expert evaluation/adjustment of the suggestions. Expert evaluation/adjustment provides feedback to the automation to improve the machine generated suggestions.

It has been noticed that different experts may end up with different conclusion when evaluating the machine generated suggestions. Certain experts may be capable of evaluating certain type of matters very well while some other matters may not belong to their core competence. Additionally or alternatively, some expert may have extensive knowledge of matters and good track record of making the right decisions when evaluating the machine generated suggestions, while some other expert may be in the beginning of their learning curve. Therefore, there may be great variation in the expert evaluation depending on the expert that is performing the evaluation. Example embodiments of present disclosure are based on collecting information about the experts and their actions and building expert profiles based on the collected information. The expert profiles may be used for weighting evaluation results of experts. In an embodiment, the expert profiles are used for weighting feedback of the experts so that for example most reliable evaluation results are S eventually used for adjusting or training the automation mechanisms. Additionally or N alternatively, the expert profiles may be used for selecting the best available expert S 25 for different cases. In an embodiment, the weighted expert feedback is used for N forming a soft target label for training the automation mechanisms, i.e. a label is z assigned with a score or probability corresponding to a specific target class. In 3 practice this could be implemented, for example, by weighted averaging of different S experts' feedback. N 30 The term expert refers herein to a person that is capable of evaluating machine generated suggestions of changes or actions for a communication network. The expert may be in early stages of learning to make educated evaluation or an expert with long experience or something in between these.

It is to be noted that in the following, profile of a single expert may be discussed, but clearly plurality of expert profiles can be built and updated correspondingly in parallel or sequentially one after another.

Fig. 1 schematically shows an example scenario according to an embodiment. The scenario shows a communication network 101 comprising a plurality of cells and base stations and other network devices, and an operations support system, OSS, 102 configured to manage operations of the communication network 101. Further, the scenario shows an automation system 111. The automation system 111 is configured to implement automated monitoring of operation of the communication network 101. The automation system 111 is operable to interact with the OSS 102 for example to receive performance data from the OSS 102 and to provide modified or new parameter values and configurations to the OSS 102 for use in the communication network 101.

Still further, the scenario of Fig. 1 shows a group of experts 103. The automation system 111 is operable to provide an interface through which one or more of the experts 103 can evaluate and adjust operation of the automation system 111 and actions automatically generated in the automation system 111.

The automation system 111 comprises an expert profiling module 112. The automation system 111 and the expert profiling system 112 are configured to interact and to implement at least some example embodiments of present o disclosure. The expert profiling module 112 is operable to store and maintain profile S database comprising expert profiles of at least some of the experts 103.

S 25 The expert profiling module 112 may be a process or software module running in N the same physical device with the automation system 111 or the expert profiling z module 112 may be implemented in a physically separate device.

2 In an embodiment of the invention the scenario of Fig. 1 operates as follows: The S automation system 111 receives performance data from the OSS 102. The data is N 30 automatically analysed in the automation system 111 and the automation system generates machine generated suggestions for optimizing operation of the communication network 101. The analysis may indicate that the communication network 101 has problems with one or more of the following: overshoot, interference, congestion, coverage etc. The machine generated suggestions may comprise suggestions of changes to be performed in the communication network 101 to overcome the detected problems e.g. by changing parameter values, modifying configuration and/or making changes in network equipment. Alternatively, the machine generated suggestions may only indicate part of the network 101 that experiences problems and requires changes. For example a cell, a base station or certain area may be indicated.

The machine generated suggestions are provided for evaluation by one or more of the experts 103. The expert may approve the suggestions or disapprove the suggestions. In case more than one expert evaluates the suggestions, the approval or disapproval of the suggestions may be based on combination of more than one evaluation. Such combination may take into account expert profiles of the different — experts and more reliable experts and/or more reliable expert profiles may be given more weight in combining the evaluation results.

Reliability of an expert depends on successfulness of expert's evaluations of machine generated suggestions. Reliability of an expert profile depends on number of evaluation results included in the profile. The reliability of an expert and the reliability of an expert profile may both be separately specified for different case types.

If an approved suggestion involves changes in the network, the automation system 111 then conveys the changes to the OSS 102 to be implemented in the N communication network 101. In a later phase effects of changes may be further N 25 evaluated by the one or more experts 103 or automatically based on e.g. statistical = analysis of performance KPIs before and after the change, and the changes may be 2 maintained or reverted to previous state based on the evaluation results. N The expert profiling module 112 collects information about results of experts’ 103 3 evaluation of the machine generated suggestions. Based on the collected ä 30 information, the expert profiling module 112 builds and updates expert profiles. The expert profiles are then used for processing further machine generated suggestions in the automation system 111. The expert profiles may be used for weighting effects that actions of respective experts may have on operation of the automation system

111. The process of collecting information about results of experts’ 103 evaluation of the machine generated suggestions may be continuous, whereby the expert profiles are continuously developed. Fig. 2 shows a block diagram of an apparatus 20 according to an embodiment. The apparatus 20 is for example a general-purpose computer or server or some other electronic data processing apparatus. The apparatus 20 can be used for implementing at least some embodiments of the invention. That is, with suitable configuration the apparatus 20 is suited for operating for example as the automation system 111 or the expert profile module 112 of foregoing disclosure. The apparatus 20 comprises a communication interface 25; a processor 21; a user interface 24; and a memory 22. The apparatus 20 further comprises software 23 stored in the memory 22 and operable to be loaded into and executed in the processor 21. The software 23 may comprise one or more software modules and can be in the form of a computer program product. The processor 21 may comprise a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a graphics processing unit, or the like. Fig. 2 shows one processor 21, but the apparatus 20 may comprise a plurality of processors. The user interface 24 is configured for providing interaction with a user of the apparatus. Additionally or alternatively, the user interaction may be implemented through the communication interface 25. The user interface 24 may comprise a S circuitry for receiving input from a user of the apparatus 20, e.g., via a keyboard, N graphical user interface shown on the display of the apparatus 20, speech S 25 recognition circuitry, or an accessory device, such as a headset, and for providing N output to the user via, e.g., a graphical user interface or a loudspeaker.

I E The memory 22 may comprise for example a non-volatile or a volatile memory, such 2 as a read-only memory (ROM), a programmable read-only memory (PROM), S erasable programmable read-only memory (EPROM), a random-access memory N 30 (RAM), a flash memory, a data disk, an optical storage, a magnetic storage, a smart card, or the like. The apparatus 20 may comprise a plurality of memories. The memory 22 may serve the sole purpose of storing data, or be constructed as a part of an apparatus 20 serving other purposes, such as processing data. The communication interface 25 may comprise communication modules that implement data transmission to and from the apparatus 20. The communication modules may comprise a wireless or a wired interface module(s) or both. The wireless interface may comprise such as a WLAN, Bluetooth, infrared (IR), radio frequency identification (RF ID), GSM/GPRS, CDMA, WCDMA, LTE (Long Term Evolution) or 5G radio module. The wired interface may comprise such as Ethernet or universal serial bus (USB), for example. The communication interface 25 may — support one or more different communication technologies. The apparatus 20 may additionally or alternatively comprise more than one of the communication interfaces

25. A skilled person appreciates that in addition to the elements shown in Fig. 2, the apparatus 20 may comprise other elements, such as displays, as well as additional circuitry such as memory chips, application-specific integrated circuits (ASIC), other processing circuitry for specific purposes and the like. Further, it is noted that only one apparatus is shown in Fig. 2, but the embodiments of the invention may equally be implemented in a cluster of shown apparatuses. Figs. 3-5 show flow diagrams illustrating example methods according to certain embodiments. The methods may be implemented in the automation system 111 and/or the expert profile module 112 of Fig. 1 and/or in the apparatus 20 of Fig. 2. The methods are implemented in a computer and do not reguire human interaction unless otherwise expressly stated. It is to be noted that the methods may however S provide output that may be further processed by humans and/or the methods may O 25 require user input to start. Different phases shown in the flow diagrams may be oO combined with each other and the order of phases may be changed except where - otherwise explicitly defined. Furthermore, it is to be noted that performing all phases : of the flow diagrams is not mandatory. 3 The method of Fig. 3 provides monitoring and controlling of a communication ä 30 network, and comprises the following phases: 301: Network data is automatically analysed and a problem is automatically identified. There is for example artificial intelligence module performing the analysis.

The identified problem may be related to one or more of the following: overshoot, interference, congestion, coverage, inefficient use of radio resources, or some other network performance problem etc.

302: The analysis results in machine generated suggestions of identified problems possibly including changes to overcome or mitigate the problems. The suggested changes may relate to changing parameter values, modifying configuration and/or making changes in network equipment.

303: The generated suggestion is evaluated by an expert. The expert may approve the suggestion, disapprove the suggestion or adjust the suggestion. 304: Information about result of the expert's evaluation of the machine generated suggestion is collected.

305: The suggestion is approved and related changes are performed in the network. It is to be noted that this is one possible choice and in some other alternative case, the suggestion may be disapproved or adjusted. That is, this phase is just one example and not in any way mandatory.

In an embodiment, the final evaluation result may be based on combination of evaluation results of more than one expert. Such combination may take into account expert profiles of the different experts and more reliable experts and/or more reliable expert profiles may be given more weight in combining the evaluation results.

2306: Effects of the changes are monitored. The purpose is to evaluate, if the changes related to the suggestion that was approved, provided desired results. For example, it may be checked if the changes solved the problem or at least improved S the situation.

N 307: The effects are evaluated by the expert. The expert may decide to maintain the = 25 changes or revert to previous state.

N I 308: Information about results of the expert's evaluation of the effects is collected.

a 00 309: Expert profile of the expert is built and updated based on the information 3 collected in phases 304 and/or 308. It is to be noted that in some cases information O is collected only in phase 304. For example, if the expert disapproves the suggestion in phase 304, phase 308 does not necessarily exist.

Further it is to be noted that the evaluation of effects in phase 307 may be automated and does not necessarily require expert evaluation.

An expert profile of an expert may comprise an identifier of the expert and information about successfulness of evaluations of the expert.

The more successful evaluations the expert performs, the more reliable the expert is considered.

For example, a suggestion approved by the expert that involves changes that are maintained after the evaluation of the effects is considered a successful evaluation.

The information about successfulness of evaluations of the expert may comprise information about whether a suggestion approved by the expert involves a change that is maintained or reverted.

Whereas, a suggestion approved by the expert involving a change that is reverted to previous state after the evaluation of the effects is considered an unsuccessful evaluation.

In an embodiment, the expert profile further comprises information about successfulness of evaluations of the expert for different case types.

For example, certain expert may be successful in evaluating overshoot cases and not so successful in evaluating congestion cases.

Still further, the information about successfulness of evaluations of the expert may comprise information about whether the expert's evaluations differ from evaluations of other experts.

For example, if a first expert would disapprove certain suggestion and three other experts would approve the same suggestion, the first expert is not considered very successful.

This example may further take into account that the three other experts should be experts that are in general considered successful or > reliable, or considered successful or reliable in the specific case type in guestion.

O In some example embodiments, the expert profile comprises information about ro 25 reliability of the expert profile and information about reliability of the expert.

S Reliability of the expert depends on successfulness of expert's evaluations of z machine generated suggestions.

Reliability of the expert profile depends on number of evaluation results included in the profile.

Reliability of the expert profile may be 2 referred to as confidence or trustworthiness of the expert profile.

For example, an ä 30 expert profile comprising information about only one or very few evaluations, is not considered very reliable irrespective of the successfulness of the evaluations.

After certain amount of evaluations have been recorded in the expert profile, the profile may be considered reliable or trustworthy.

For example 50-200 evaluation cases (of same case type) may be required for establishing reliability of the expert profile.

In an alternative embodiment, reliability of the expert profile is first built before the expert profile is taken into use.

In such case, reliability of expert profiles need not be considered when using the expert profiles.

Expert profile and reliability of the expert profile of a new expert may be built for example based on the new expert evaluating machine generated suggestions that have already been evaluated by some other expert (or experts) or that are concurrently being evaluated by some other expert (or experts). Expert profile of the new expert is then built based on comparison of the evaluations of the new expert to evaluations of the other experts.

This is an option for bringing onboard new expert to an already functioning arrangement.

If there are no existing experts (in the whole arrangement or for a certain new case type), this option is not available and some other approach may be taken to build the very first expert profiles.

They may be for example manually generated.

Reliability of an expert profile may be separately specified for different case types, which further improves profiling of the experts so that best available expert can be selected for each case.

In some cases, the expert(s) may agree with the need for a change related to a machine generated suggestion, but the change is still not taken for other reasons such as cost of making the change.

In such cases, the experts can be evaluated against one another the same way as in the case the suggestion is disapproved.

The feedback for the automation system can be then given as if the suggestion S would have been approved. ro 25 310: The expert profile is rated in relation to other expert profiles.

There may be an S expert database storing a plurality of expert profiles and the expert profiles in the z database may be rated based on information about successfulness of evaluations N of the experts.

Based on the rating, the best available expert (most reliable expert 3 among the experts) may be chosen.

S 30 Additionally or alternatively, the expert profiles may be categorized into different classes based on information about successfulness of evaluations of the experts.

That is, there is no need to perform comparison to individual other expert profiles.

There may be for example a class of best performing experts that requires certain percentage of the evaluations of the expert to be successful. For example 75-95% success rate may be required. 50%-75% success rate could be categorized into a class having mediocre success. Less than 50% success rate could be categorized into a class having insufficient success. Clearly these are only examples and there may be any suitable number of classes with any suitable requirements. The rating and categorization may take into account reliability of the expert profile in addition to the successfulness (i.e. reliability) of the experts. In this way, for example an expert with a single successful evaluation is not rated best or categorized into the best performing class. Instead good ranking requires plurality of successful evaluations. Still further, the rating and categorization may be different for different case types. An expert profile module according to some embodiments may implement phases 304, and 308-310. Other phases discussed in Fig. 3 provide context in which the expert profile module is operating. The method of Fig. 4 provides examples of using the expert profiles, and comprises the following phases: 401: The expert profiles built e.g. in phase 309 of Fig. 3 are being used for processing further machine generated suggestions. That is, the expert profile module learns and builds expert profiles that are then used when processing later appearing problems. 402: Feedback is provided to generation of the machine generated suggestions N based on the results of the experts' evaluation to adjust the generation of the further N machine generated suggestions. The feedback is weighted based on the respective = 25 expert profiles.

N I For example, if the profile of the expert indicates that the expert has successfully N evaluated plurality of similar cases, the expert feedback may be fully taken into 3 account. Whereas, if the profile of the expert indicates that the expert has not N evaluated any similar cases before, the expert feedback may be fully ignored.

N Additionally or alternatively, feedback of the expert who is rated most successful or feedback of the experts categorized into best performing class may be fully taken into account (or is taken into account with a larger weight), while feedback of other experts may be given smaller weight. 404: The expert profiles are used for choosing an expert for evaluating a further machine generated suggestion. For example, the best available expert for certain case may be chosen based on the expert profiles. 405: The expert profiles are used for choosing an additional expert for evaluating a further machine generated suggestion. Evaluation of an additional expert may be used for example for analyzing evaluation of a junior expert with less experience and short or non-existing track record of evaluations. The evaluations of the junior expert and the additional expert may be compared to check if evaluation of the junior expert can be considered successful. 406: In some example embodiments evaluation results of different experts may be combined to obtain final evaluation result. In such case, the expert profiles may be used to weight the evaluation results of individual experts in the combined evaluation result. All phases of Fig. 4 may employ rating and/or categorization of expert profiles performed in phase 310 of Fig. 3. The method of Fig. 5 provides another example of monitoring and controlling of a communication network, and comprises the following phases: 301: Network data is automatically analysed and a problem is automatically identified. There is for example artificial intelligence module performing the analysis. o The identified problem may be related to one or more of the following: overshoot, O interference, congestion, coverage, inefficient use of radio resources, etc. 3 302: The analysis results in machine generated suggestion of identified problems possibly including changes to overcome the problems. The suggested changes may x late to changi t | difyi fi ti d/ ki = relate to changing parameter values, modifying configuration and/or making 00 changes in network eguipment.

LO S 503: The generated suggestion is evaluated by a first expert. The expert may

O NUNN N approve the suggestion, disapprove the suggestion or adjust the suggestion.

504: Information about result of the first expert's evaluation of the machine generated suggestion is collected.

505: The first expert disapproves the suggestion. It is to be noted that this is one possible choice and in some other alternative case, the first expert may approve or adjust the suggestion.

506: The generated suggestion is submitted for evaluation by a second expert. This may be responsive to the first expert disapproving the suggestion or responsive to the first expert being a less experienced expert. Additionally or alternatively, certain type of suggestions (e.g. suggestions involving certain changes) may be by default submitted for evaluation by more than one expert. For example expensive network changes may require evaluation by more than one expert.

507: Information about result of the second expert's evaluation of the machine generated suggestion is collected. 508: Expert profile of the first expert is built and updated based on the information collected in phases 504 and 507. It is to be noted that in some cases information may be collected only in phase 504.

An expert profile module according to some embodiments may implement phases 504, and 507-508. Other phases discussed in Fig. 5 provide context in which the expert profile module is operating.

— Without in any way limiting the scope, interpretation, or application of the appended claims, a technical effect of one or more of the example embodiments disclosed herein is improvement in automated monitoring and control arrangements as N reliability (or successfulness) of the experts evaluating the machine generated N suggestions is automatically taken into account and effects of experts' actions in = 25 adjusting the operation of the automation mechanisms depend on reliability of the - expert. More reliable experts will have stronger effect on development of the E automation mechanisms and thereby development of the automation mechanisms 2 is likely to advance into right direction.

LO N Another technical effect of one or more of the example embodiments is that use of N 30 junior experts is possible as the experts’ reliability is automatically taken into account.

If desired, the different functions discussed herein may be performed in a different order and/or concurrently with each other. Furthermore, if desired, one or more of the before-described functions may be optional or may be combined Various embodiments have been presented. It should be appreciated that in this document, words comprise, include and contain are each used as open-ended expressions with no intended exclusivity.

The foregoing description has provided by way of non-limiting examples of particular implementations and embodiments a full and informative description of the best mode presently contemplated by the inventors for carrying out the invention. It is however clear to a person skilled in the art that the invention is not restricted to details of the embodiments presented in the foregoing, but that it can be implemented in other embodiments using equivalent means or in different combinations of embodiments without deviating from the characteristics of the invention.

Furthermore, some of the features of the afore-disclosed example embodiments may be used to advantage without the corresponding use of other features. As such, the foregoing description shall be considered as merely illustrative of the principles of the present invention, and not in limitation thereof. Hence, the scope of the invention is only restricted by the appended patent claims.

O QA O N

LÖ <Q

O N

I a a 00

O LO LO O QA O N

Claims

1. A computer implemented method of monitoring and controlling a communication network (101), the method comprising collecting (304, 308, 504, 507) information about results of experts' evaluation of machine generated suggestions, wherein the machine generated suggestions relate to making changes in the communication network (101); building (309, 508) and updating expert profiles based on the collected information; and using (401-406) the expert profiles for processing further machine generated suggestions.

2. The method of claim 1, wherein the expert profile of an expert comprises an identifier of the expert and information about successfulness of evaluations of the expert.

3. The method of claim 2, wherein the expert profile further comprises information about successfulness of evaluations of the expert for different case types.

4 The method of claim 2 or 3, wherein the information about successfulness of o evaluations of the expert comprises information about whether a change approved

QA S by the expert is maintained or reverted.

LÖ <Q

N - 5. The method of any one of claims 2-4, wherein the information about a = 25 — successfulness of evaluations of the expert comprises information about whether 2 the expert's evaluations differ from evaluations of other experts.

S

QA

O

N

6. The method of any preceding claim, wherein the expert profile of an expert comprises information about reliability of the expert profile.

7. The method of any preceding claim, further comprising storing a plurality of expert profiles to form an expert database; rating (310) the expert profiles in relation to other expert profiles in the expert database based on information about successfulness of evaluations of the experts and/or based on reliability of the expert profiles; and using the ratings of the expert profiles when using the expert profiles for processing further machine generated suggestions.

8. The method of any preceding claim, wherein the machine generated suggestions specify concrete changes to be performed in the communication network.

9. The method of any one of claims 1-7, wherein the machine generated — suggestions specify a part of network where changes are needed.

10. The method of any preceding claim, wherein using the expert profiles comprises weighting evaluation results of experts based on the respective expert profiles.

S

11. The method of any preceding claim, further comprising providing feedback to N generation of the machine generated suggestions based on the results of the

LO ? experts’ evaluation to adjust the generation of the further machine generated

O N suggestions, wherein using the expert profiles comprises weighting (402) the E: 25 feedback based on the respective expert profiles.

00

O

LO

S S

12. The method of any preceding claim, wherein using the expert profiles comprises choosing (404) an expert for evaluating a further machine generated suggestion based on the expert profiles.

13. The method of any preceding claim, wherein using the expert profiles comprises choosing (405) an additional expert for evaluating a further machine generated suggestion based on the expert profiles.

14. An apparatus (20, 111, 112) comprising a processor (21), and a memory (22) including computer program code; the memory and the computer program code configured to, with the processor, cause the apparatus to perform the method of any one of claims 1-12.

15. A computer program comprising computer executable program code (23) which when executed by a processor causes an apparatus to perform the method of any one of claims 1-12.

O

QA

O

N

LÖ <Q

O

N

I = 00

O

LO

O

QA

O

N