WO2022058647A1

WO2022058647A1 - Evaluating effect of a change made in a communication network

Info

Publication number: WO2022058647A1
Application number: PCT/FI2021/050582
Authority: WO
Inventors: Petteri LUNDÉN; Adriana CHIS
Original assignee: Elisa Oyj
Priority date: 2020-09-21
Filing date: 2021-08-30
Publication date: 2022-03-24
Also published as: FI20205910A1; EP4214948A1

Abstract

A computer implemented method for evaluating effect of a change made in a communication network. The method is performed by observing (210) values of a characterization indicator over a first time period without the change and over a second time period with the change; adjusting (211) at least one of the first and second time periods based on values of the characterization indicator to obtain first and second adjusted time periods; and comparing (212) values of a performance indicator over the first adjusted time period with values of the performance indicator over the second adjusted time period to evaluate the effect of the change made in the communication network.

Description

EVALUATING EFFECT OF A CHANGE MADE IN A COMMUNICATION NETWORK

TECHNICAL FIELD

The present application generally relates to automated evaluation of effects of changes, such as parameter changes or other changes, made in cellular communication networks.

BACKGROUND

This section illustrates useful background information without admission of any technique described herein representative of the state of the art.

Cellular communication networks comprise a plurality of cells serving users of the network. When users of the communication network move in the area of the network, connections of the users are seamlessly handed over between cells of the network. In order to provide good quality of service for users of the network, different parts of the network need to operate as intended.

There are various network parameters that affect operation of individual cells of the network and/or the network in larger scale. For example, due to network topology and usage of the network evolving or for other reasons there is constant need to change (values of) various parameters or to make other changes to optimize operation of the cells of the network. Such parameters that are changed comprise for example antenna tilt, transmission power, handover parameters and plurality of other parameters.

Impact of a parameter change may be evaluated by comparing performance indicators before and after the change. A challenge in such comparison is that there are other factors that affect the performance, too. If the time period over which the evaluation is done is long, the performance indicators may include seasonal effects of long-term changes. Shorter time periods, on the other hand, make analysis more difficult as it is more difficult to average out noise due to inherent uncertainty in network load, traffic patterns, weather conditions etc. Now there is provided a new method of monitoring impact of parameter changes.

SUMMARY

The appended claims define the scope of protection. Any examples and technical descriptions of apparatuses, products and/or methods in the description and/or drawings not covered by the claims are presented not as embodiments of the invention but as background art or examples useful for understanding the invention.

According to a first example aspect there is provided a computer implemented method for evaluating effect of a change made in a communication network. The method comprises observing values of a characterization indicator over a first time period without the change and over a second time period with the change; adjusting at least one of the first and second time periods based on values of the characterization indicator to obtain first and second adjusted time periods; and comparing values of a performance indicator over the first adjusted time period with values of the performance indicator over the second adjusted time period to evaluate the effect of the change made in the communication network.

In some example embodiments, adjusting at least one of the first and second time periods comprises filtering out one or more time periods associated with a deviating characterization indicator value.

In some example embodiments, the deviating characterization indicator value is a value that substantially deviates from average characterization indicator value.

In some example embodiments, the deviating characterization indicator value is a value that is present only in first values of the characterization indicator observed during the first time period or in second values of the characterization indicator observed during the second time period.

In some example embodiments, the method further comprises repeatedly making and reverting the change to obtain a plurality of first sub periods forming the first time period and a plurality of second sub periods forming the second time period, wherein said adjusting at least one of the first and second time periods comprises filtering out at least one of the first or second sub periods.

In some example embodiments, adjusting at least one of the first and second time periods comprises adjusting the first and second time periods such that distribution of values of the characterization indicator is substantially similar over the first adjusted time period and over the second adjusted time period.

In some example embodiments, adjusting at least one of the first and second time periods comprises extending the second time period until the values of the characterization indicator observed during the second time period substantially cover the range of the values of the characterization indicator observed during the first time period; or vice versa, extending the first time period until the values of the characterization indicator observed during the first time period substantially cover the range of the values of the characterization indicator observed during the second time period.

In some example embodiments, the change made in the communication network is related to one or more of power save, performance optimization, increasing capacity, load balancing and solving a performance problem.

In some example embodiments, the performance indicator is spectral efficiency or user throughput and the characterization indicator is traffic volume or number of active users. Clearly these are only example and also other indicators can be used.

In some example embodiments, the method further comprises, responsive to the result of the evaluation of the change made in the communication network, keeping the change or reverting the change.

According to a second example aspect of the present invention, there is provided an apparatus comprising a processor and a memory including computer program code; the memory and the computer program code configured to, with the processor, cause the apparatus to perform the method of the first aspect or any related embodiment.

According to a third example aspect of the present invention, there is provided a computer program comprising computer executable program code which when executed by a processor causes an apparatus to perform the method of the first aspect or any related embodiment.

According to a fourth example aspect there is provided a computer program product comprising a non-transitory computer readable medium having the computer program of the third example aspect stored thereon.

According to a fifth example aspect there is provided an apparatus comprising means for performing the method of the first aspect or any related embodiment.

Any foregoing memory medium may comprise a digital data storage such as a data disc or diskette, optical storage, magnetic storage, holographic storage, opto- magnetic storage, phase-change memory, resistive random access memory, magnetic random access memory, solid-electrolyte memory, ferroelectric random access memory, organic memory or polymer memory. The memory medium may be formed into a device without other substantial functions than storing memory or it may be formed as part of a device with other functions, including but not limited to a memory of a computer, a chip set, and a sub assembly of an electronic device.

Different non-binding example aspects and embodiments have been illustrated in the foregoing. The embodiments in the foregoing are used merely to explain selected aspects or steps that may be utilized in different implementations. Some embodiments may be presented only with reference to certain example aspects. It should be appreciated that corresponding embodiments may apply to other example aspects as well.

BRIEF DESCRIPTION OF THE FIGURES

Some example embodiments will be described with reference to the accompanying figures, in which:

Fig. 1 is a graph showing performance before and after a parameter change;

Fig. 2A schematically shows an example scenario according to an example embodiment;

Fig. 2B shows a block diagram of an apparatus according to an example embodiment; and Fig. 2C shows a flow diagram illustrating example methods according to certain embodiments; and

Figs. 3-6 show graphs illustrating some example cases.

DETAILED DESCRIPTION

Example embodiments of the present invention and its potential advantages are understood by referring to Figs. 1 through 6 of the drawings. In the following description, like reference signs denote like elements or steps.

Example embodiments of the invention provide evaluation of effects of a change made in a communication network for the purpose of controlling the communication network. The change may be a network parameter change or some other change in a cell, sector or base station site. The parameter that is changed may be for example antenna tilt, transmission power, handover parameter or some other parameter that may be adjusted in a communication network. In general, effects of the parameter change can be seen in behaviour of one or more performance indicators. If the parameter change improves operation of the network, the values of the performance indicators should improve. However, it is not always clear whether changes in performance indicator values are a consequence of a particular parameter change or a consequence of something else, such as changes in weather, changes in network usage, changes in user behaviour etc.

Fig. 1 is a graph showing performance before and after a parameter change. The graph shows 4 different performance indicator values 151 -154 as a function of time. The performance indicators may relate to spectral efficiency, signal level, throughput, number of dropped calls or other performance indicators available in a communication network. Line 150 indicates point of time when a parameter value is changed. Before the point of time 150, a first value is used and after the point of time 150 a second value is used. It can be clearly seen that comparing any one of the performance indicators before and after the point of time 150 is not straightforward as there is no clearly visible difference in the performance graphs.

Based on this, evaluation of effects of a change by comparing of performance before and after the change is not a straightforward task. Fig. 2A schematically shows an example scenario according to an embodiment. The scenario shows a communication network 101 comprising a plurality of cells and base stations and other network devices, and an operations support system, OSS, 102 configured to manage operations of the communication network 101. Further, the scenario shows an automation system 111. The automation system 111 is configured to implement automated monitoring of operation of the communication network 101 . The automation system 111 is operable to interact with the OSS 102 for example to receive performance data from the OSS 102 and to provide modified or new parameter values and configurations to the OSS 102 for use in the communication network 101.

The automation system 111 is configured to implement at least some example embodiments of present disclosure.

In an embodiment of the invention the scenario of Fig. 2A operates as follows: The automation system 111 receives performance data comprising values of performance indicators from the OSS 102. The automation system gathers the performance data associated with a first time period before a change is implemented in the communication network and with a second time period after the change. The change may comprise changing one or more parameter values, modifying configuration and/or making changes in network equipment (such as upgrading hardware or software, or adding new capacity by deploying new cells or carriers).

The performance data is automatically analysed in the automation system 111 to evaluate effects of the change made in the communication network. Various embodiments of present disclosure provide that characterization indicator values are used for selecting which performance indicator values to compare with each other in this analysis. The results of the analysis may be provided for further automated processes running in the automation system 111 or shown on a display or otherwise output to a user.

The analysis may be automatically or manually triggered. The analysis may be performed in association with all changes implemented in the communication network or in association with some selected changes.

Fig. 2B shows a block diagram of an apparatus 20 according to an embodiment. The apparatus 20 is for example a general-purpose computer or server or some other electronic data processing apparatus. The apparatus 20 can be used for implementing at least some embodiments of the invention. That is, with suitable configuration the apparatus 20 is suited for operating for example as the automation system 111 of foregoing disclosure.

The apparatus 20 comprises a communication interface 25; a processor 21 ; a user interface 24; and a memory 22. The apparatus 20 further comprises software 23 stored in the memory 22 and operable to be loaded into and executed in the processor 21. The software 23 may comprise one or more software modules and can be in the form of a computer program product.

The processor 21 may comprise a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a graphics processing unit, or the like. Fig. 2 shows one processor 21 , but the apparatus 20 may comprise a plurality of processors.

The user interface 24 is configured for providing interaction with a user of the apparatus. Additionally or alternatively, the user interaction may be implemented through the communication interface 25. The user interface 24 may comprise a circuitry for receiving input from a user of the apparatus 20, e.g., via a keyboard, graphical user interface shown on the display of the apparatus 20, speech recognition circuitry, or an accessory device, such as a headset, and for providing output to the user via, e.g., a graphical user interface or a loudspeaker.

The memory 22 may comprise for example a non-volatile or a volatile memory, such as a read-only memory (ROM), a programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), a random-access memory (RAM), a flash memory, a data disk, an optical storage, a magnetic storage, a smart card, or the like. The apparatus 20 may comprise a plurality of memories. The memory 22 may serve the sole purpose of storing data, or be constructed as a part of an apparatus 20 serving other purposes, such as processing data.

The communication interface 25 may comprise communication modules that implement data transmission to and from the apparatus 20. The communication modules may comprise a wireless or a wired interface module(s) or both. The wireless interface may comprise such as a WLAN, Bluetooth, infrared (IR), radio frequency identification (RF ID), GSM/GPRS, CDMA, WCDMA, LTE (Long Term Evolution) or 5G radio module. The wired interface may comprise such as Ethernet or universal serial bus (USB), for example. The communication interface 25 may support one or more different communication technologies. The apparatus 20 may additionally or alternatively comprise more than one of the communication interfaces 25.

A skilled person appreciates that in addition to the elements shown in Fig. 2B, the apparatus 20 may comprise other elements, such as displays, as well as additional circuitry such as memory chips, application-specific integrated circuits (ASIC), other processing circuitry for specific purposes and the like. Further, it is noted that only one apparatus is shown in Fig. 2B, but the embodiments of the invention may equally be implemented in a cluster of shown apparatuses.

Fig. 2C shows a flow diagram illustrating example methods according to certain embodiments. The methods may be implemented in the automation system 111 of Fig. 2A and/or in the apparatus 20 of Fig. 2B. The methods are implemented in a computer and do not require human interaction unless otherwise expressly stated. It is to be noted that the methods may however provide output that may be further processed by humans and/or the methods may require user input to start. Different phases shown in the flow diagrams may be combined with each other and the order of phases may be changed except where otherwise explicitly defined.

The methods of Fig. 2C provide evaluating effects of a change made in a communication network. In an embodiment the change made in the communication network is a network parameter change in a cell of interest or in a sector of interest or in a base station site of interest. Additionally or alternatively, the change may be related to one or more of power/energy save, performance optimization, increasing capacity, load balancing and solving a performance problem. The change may be for example a change in power/energy saving schedule, antenna tilt angle, handover parameter, load balancer or some other configuration parameter of the communication network.

The method of Fig. 2C comprises the following phases:

210: Values of a characterization indicator are observed over a first time period and over a second time period. During the first time period the change has not been made yet, i.e. the first time period is without the change. During the second time period the change has been made, i.e. the second time period is with the change (e.g. with a new configuration or parameter setting).

In an embodiment, such characterization indicator is chosen that is substantially invariant to the change made in the communication network. The characterization indicator may be related to load in the communication network, for example.

The first and second time periods may be equally long or their lengths may differ from each other. In some embodiments, the first and second time periods are multiples of an hour length sub periods. In some embodiments, the first and second time periods are multiples of 24 hour time periods. By using a 24 hour time period or multiples thereof, one achieves that hourly variation in network usage is covered.

At the same time with observing the values of the characterization indicator, also values of a performance indicator are observed, but they are further processed only in later phases.

211 : At least one of the first and second time periods is adjusted based on values of the characterization indicator to obtain first and second adjusted time periods. Adjusting the time periods may refer e.g. to filtering out or ignoring some shorter time periods within the first and second time periods. In this way one obtains time periods that are used for analyzing effects of a change made in the network based on values of a performance indicator.

In some embodiments, the characterization indicator is used for choosing such first and second adjusted time periods that represent normal operating conditions in the communication network. For example unusual peaks in network load or time periods with unusual external disturbance may be filtered out from the adjusted time periods.

In some embodiments, the characterization indicator is used for choosing such first and second adjusted time periods that represent substantially equal operating conditions in the communication network. Le., network operation is substantially equal over the first adjusted time period and over the second adjusted time period.

In some embodiments, one or more time periods associated with a deviating characterization indicator value are filtered out. The deviating characterization indicator value may be for example a value that substantially deviates from average characterization indicator value or from usual range/value of characterization indicator values. E.g. one or more time periods during which value of the characterization indicator is at maximum and/or minimum can be filtered out. In this way, time periods with anomalous or extreme values of characterization indicator are filtered out. Such time periods may relate to sudden abnormal behaviour in the network and therefore filtering them out may improve the evaluation of the effects of the change. In some embodiments, the anomalous or extreme values of characterization indicator are filtered out only if the anomalous or extreme values are present only in one of the first and second time periods, and if they are present in both time periods they are kept. In a practical example, first values of the characterization indicator observed during the first time period and second values of the characterization indicator observed during the second time period are compared with each other. If there are deviating characterization indicator values that are present only in the first values of the characterization indicator or in the second values of the characterization indicator, time periods associated with such deviating values are filtered out from the first or the second time period to obtain the first and second adjusted time periods. However, if the first values of the characterization indicator and the second values of the characterization indicator are substantially similar, filtering is not necessarily needed and the first and second adjusted time periods may be identical with the first and second time periods.

Further details about example implementations of the time period adjustment are discussed in more detail in connection with Figs. 3-6.

212: Values of a performance indicator over the first adjusted time period are compared with values of the performance indicator over the second adjusted time period to evaluate the effect of the change made in the communication network.

The evaluation of the effect of the change made in the communication network can be performed by suitable mathematical calculation such as subtraction or division. This can be interpreted as an absolute or a relative change in the performance indicator values.

For the sake of clarity, it is noted that initially values of both the characterization indicator and the performance indicator are concurrently observed over the first time period and the second time period. Then only values of the performance indicator observed over the first adjusted time period and the second adjusted time period are used in phase 212. That is, some values of the performance indicator may be excluded from the comparison of phase 212 based on the values of the characterization indicator.

213: Based on the evaluation result the change is kept or reverted. More specifically, if the evaluation indicates improvement in performance, the change is kept and possibly also a new change in same direction may be made. If the evaluation indicates degradation or insignificant improvement in performance, the change may be reverted or other corrective action could be taken to address the performance problem.

In general, the purpose of the characterization indicator is to identify and filter out those changes in network performance that are likely not caused by the new configuration or parameter setting or at least not fully caused by the new configuration or parameter setting. There may exist for example performance changes that are caused by other events in the network (e.g. temporary increase in number of users). In an embodiment, the aim is to select the characterization indicator so that it is as invariant as possible to the new configuration or parameter setting so that the characterization indicator would be usable for identifying changes in network performance that are not caused by the new configuration or parameter setting. Unusual values of the characterization indicator suggest that it is possible or even likely that the situation in the network has been unusual and thus the performance indicator values collected from the corresponding time period may not give a representative or comparable picture of the performance in typical operating conditions. In some example embodiments, the characterization indicator is traffic volume or number of active users. For example, the traffic volume or the number of users or connections over a time period and over an area covering multiple cells depends primarily on the usage pattern and only marginally on parameter settings such as antenna tilt angle, power/energy saving schedule, handover parameters, load balancer configuration or some other configuration. In some cases the configuration change being evaluated is such that it is expected not to affect the cell borders, such as for example many types of software updates, in which case characterization indicator could be a signal measurement such as RSRP (Reference Signal Received Power), RSRQ (Reference Signal Received Quality), SINR (Signal to Interference plus Noise Ratio), TA (Timing Advance) or CQI (Channel Quality Indicator) data that can give more accurate view of the user locations within a cell and thus can even more accurately characterize the network state during a time period. More than one characterization indicator type may be taken into account to provide more extensive evaluation.

The performance indicator may relate for example to spectral efficiency, user throughput, SINR values, RSRP values, signal strengths or to some other parameter that is in general affected by the change made in the network. More than one performance indicator type may be taken into account to provide more extensive evaluation.

The characterization indicator values and the performance indicator values may be received e.g. from OSS of the communication network or from other source. All values may be received concurrently or the values may be received in smaller sets. The characterization indicator values and the performance indicator values may provide absolute values or they may be bin structured (e.g. histograms). It is possible that only part of the received values are analysed, or that the received values are analysed in parts for evaluating different effects. The analysis may be directed to certain percentile, such as for example 10^th, 50^th or 90^th percentile.

Figs. 3-6 show graphs illustrating some example cases. Figs. 3-5 show characterization indicator (referred to as a characterization KPI, key performance indicator) and performance indicator (referred to as a performance KPI) as a function of time. In the discussed examples, values of a parameter is changed and a characterization indicator and a performance indicator are monitored.

Fig. 3 shows a first time period 301 with a first parameter setting and a second time period 302 with a second parameter setting. The characterization indicator graph of Fig. 3 shows that value of the characterization indicator deviates from usual range/value or from average range/value at a first point 303 and a second point 304. At the first point 303, the characterization indicator is at its minimum and at the second point 304 the characterization indicator is at its maximum. These time periods 311 and 314 with minimum and maximum values are differing from the rest of the time periods substantially and are therefore filtered out. In the shown example, a time period 311 around the first point 303 and a time period 314 around the second point 304 are filtered out from the first and second time periods 301 , 302 to obtain first and second adjusted time periods. In this example, the first adjusted time period is formed of remaining time periods 310 and 312 of the first time period 301 and the second adjusted time period is formed of remaining time periods 313 and 315 of the second time period 302.

Evaluation of the parameter change in the example of Fig. 3 is then performed based on values of the performance indicator over the time periods 310, 312 forming the first adjusted time period and the time periods 313, 315 forming the second adjusted time period.

Fig. 4 shows an example, wherein the first and second time periods discussed in phase 210 of Fig. 2C are formed by repeatedly making and reverting the change to obtain a plurality of first sub periods forming the first time period and a plurality of second sub periods forming the second time period. Fig. 4 shows sub periods 402,

404, 406, 408, 410, 412 forming the first time period, and sub periods 403, 405, 407,

409, 411 , 413 forming the second time period.

The characterization indicator graph of Fig. 4 shows that value of the characterization indicator deviates from usual range/value or from average range/value at a first point 420 during sub period 403 and a second point 421 during sub period 409. The sub periods 403 and 409 are filtered out from the first and second time periods to obtain first and second adjusted time periods. Thereby, in this example, the first adjusted time period is formed of the sub periods 402, 404, 406, 408, 410, 412 and the second adjusted time period is formed of the sub periods

405, 407, 411 , 413.

Evaluation of the parameter change in the example of Fig. 4 is then performed based on values of the performance indicator over the sub periods 402, 404, 406, 408,

410, 412 forming the first adjusted time period and the sub periods 405, 407, 411 , 413 forming the second adjusted time period.

Fig. 5 shows an example, wherein the second time period is extended until the value of the characterization indicator during the second time period reaches maximum of the value of the characterization indicator during the first time period. In the shown example, the change is repeatedly made and reverted and values of the characterization indicator and values of the performance indicator are first collected over a plurality of sub periods 402, 404, 406, 408 forming the first time period and for a plurality of sub periods 403, 405, 407 forming the second time period in a similar manner to the example of Fig. 4.

The characterization indicator values and the performance indicator values collected with the change and without the change are analysed.

The characterization indicator graph of Fig. 5 shows that value of the characterization indicator deviates from usual range/value or from average range/value at a first point 502 during sub period 402, which is a sub period of the first time period, i.e. without the change. At time T it is concluded that there is sufficient amount of data to compare values of the performance indicator with and without the change, except that there is no sub period of the second time period, i.e. with the change, having a substantially similar high value as the sub period 402 of the first time period. Consequently the second time period with the change is extended for an extended time period 510 until the characterization indicator reaches at second point 503 a substantially similar high value as in the first point 502.

Thereby, in this example, the first adjusted time period is formed of the sub periods

402, 404, 406, 408 and the second adjusted time period is formed of the sub periods

403, 405, 407, 510. Evaluation of the parameter change in the example of Fig. 5 is then performed based on values of the performance indicator over the sub periods 402, 404, 406, 408 forming the first adjusted time period and the sub periods 403, 405, 407, 510 forming the second adjusted time period.

In the example of Fig. 5, there is no need to ignore any monitored performance indicator values as time periods are adjusted so that comparable results are achievable while taking all monitored performance indicator values into account.

It is to be noted that the example of Fig. 5 may be correspondingly applied to extending the first time period and/or to a low or minimum value of the characterization indicator or to some other value or value range that is not included in one of the first or second time periods. Furthermore it is to be noted that herein reaching the high/low value does not require reaching exactly the same values. Instead it suffices to reach about the same level with a suitable margin. In some embodiments, the aim is to adjust the first and second time periods such that distribution of values of the characterization indicator is substantially similar over the first adjusted time period and the second adjusted time period. The values of the performance indicator over the first adjusted time period and the second adjusted time period can then be compared e.g. hour by hour with substantially similar corresponding characterization indicators.

Fig. 6 shows some example histograms of distribution of the characterization indicator. Scenario a) shows distribution of the characterization indicator without time period adjustment and scenario b) shows distribution of the characterization indicator with time period adjustment.

In scenario a), graph 601 shows distribution of the characterization indicator with a first parameter value and graph 602 shows distribution of the characterization indicator with a second parameter value. It can be seen that distribution of the characterization indicator changes substantially from the first time period (first parameter value) to the second time period (second parameter value).

In scenario b), graph 603 shows adjusted distribution of the characterization indicator with a first parameter value and graph 604 shows adjusted distribution of the characterization indicator with a second parameter value. It can be seen that the adjusted distribution of the characterization indicator is substantially similar with the first parameter value and the second parameter value. This is achieved by filtering and/or rearranging sub periods of the first time period with the first parameter value and the second time period with the second parameter value. The performance indicator values corresponding to the filtered and/or rearranged sub periods may then be compared to evaluate effect of changing from the first parameter value to the second parameter value.

In some embodiments, the characterization indicator may be an aggregated value determined over a plurality of cells instead of taking values only form one cell. For example if a change is made in a first cell, the characterization indicator may be based on characterization indicator values from a cluster of cells near the first cell, such as the first cell and its neighbor cells. This may be beneficial for example in cases where the change made in the network affects cell borders and may shift some of the cell load to neighboring cells. That is, characterization indicator may show changes in individual cells but in a larger area or cluster of cells, the aggregated characterization indicator is likely to remain substantially similar.

In some embodiments, the performance indicator may be an aggregated value determined over a plurality of cells instead of taking values only form one cell. For example if a change is made in a first cell, the performance indicator may be based on performance indicator values from a cluster of cells near the first cell, such as the first cell and its neighbor cells. This may be beneficial for example in cases where the change made in the network affects cell borders and may shift some of the cell load to neighboring cells. Performance can also be aggregated over a larger number of cells. In this way it may be possible to capture aggregate impact that is reflected on the performance of multiple cells. (Instead of analyzing the cells individually.) The set of cells could be the same as for the characterization indicator, but not necessarily. It is possible e.g. to evaluate characterization indicator over multiple cells, but evaluate performance impact in one cell only.

In an embodiment, a decision on scheduling a change in the network can be made based on predicted characterization indicator. For example, if values of the performance indicator corresponding to certain value range of the characterization indicator are missing from the second adjusted time period (i.e. with the change being made) and it is predicted that during certain time period the characterization indicator is within that range, then it can be scheduled to have the change effective during that certain time period in order to collect the missing values of the performance indicator. In this way, complete and more accurate comparison of performance indicator values may be achieved.

The same applies respectively to a situation where the values of the performance indicator are missing from the first adjusted time period.

In an embodiment, in cases where the characterization indicator is substantially similar over plurality of sub periods of the first and second adjusted time periods, the comparison of the values of the performance indicator prioritizes comparison of data collected at the same time of day and/or same weekday. In an example implementation, a distance function is applied to provide this effect. Such distance function is used that takes into account both difference in values of the characterization indicator as well as difference in time of day/weekday. This could be for example cosine similarity or Euclidean distance between the values (after normalizing).

Without in any way limiting the scope, interpretation, or application of the appended claims, a technical effect of one or more of the example embodiments disclosed herein is improved or at least an alternative evaluation of effects of a change in a communication network. By using a characterization indicator for adjusting the time periods taken into account in evaluation of the effect of a change, more reliable comparison may be achieved. The use of the characterization indicator helps in selecting comparison periods so that the periods are as similar as possible with regard to external effects whereby it may be possible to improve accuracy of the comparison. Additionally or alternatively, effects of seasonality and trends may be mitigated.

If desired, the different functions discussed herein may be performed in a different order and/or concurrently with each other. Furthermore, if desired, one or more of the before-described functions may be optional or may be combined.

Various embodiments have been presented. It should be appreciated that in this document, words comprise, include and contain are each used as open-ended expressions with no intended exclusivity.

The foregoing description has provided by way of non-limiting examples of particular implementations and embodiments a full and informative description of the best mode presently contemplated by the inventors for carrying out the invention. It is however clear to a person skilled in the art that the invention is not restricted to details of the embodiments presented in the foregoing, but that it can be implemented in other embodiments using equivalent means or in different combinations of embodiments without deviating from the characteristics of the invention.

Furthermore, some of the features of the afore-disclosed example embodiments may be used to advantage without the corresponding use of other features. As such, the foregoing description shall be considered as merely illustrative of the principles of the present invention, and not in limitation thereof. Hence, the scope of the invention is only restricted by the appended patent claims.

Claims

1 . A computer implemented method for evaluating effect of a change made in a communication network (101 ) for deciding whether to keep the change or to revert the change, characterized by observing (210) values of a characterization indicator over a first time period without the change and over a second time period with the change; adjusting (211 ) at least one of the first and second time periods based on values of the characterization indicator to obtain first and second adjusted time periods; and comparing (212) values of a performance indicator over the first adjusted time period with values of the performance indicator over the second adjusted time period to evaluate the effect of the change made in the communication network.

2. The method of claim 1 , wherein adjusting at least one of the first and second time periods comprises using the characterization indicator to choose such first and second adjusted time periods that represent normal operating conditions in the communication network.

3. The method of claim 1 or 2, wherein adjusting at least one of the first and second time periods comprises using the characterization indicator to choose such first and second adjusted time periods that represent substantially equal operating conditions in the communication network.

4. The method of claim 1 , 2 or 3, wherein adjusting at least one of the first and second time periods comprises filtering out one or more time periods (311 , 314, 403, 409) associated with a deviating characterization indicator value.

5. The method of claim 4, wherein the deviating characterization indicator value is a value that substantially deviates from average characterization indicator value.

6. The method of claim 4, wherein the deviating characterization indicator value is a value that is present only in first values of the characterization indicator observed during the first time period or in second values of the characterization indicator observed during the second time period.

7. The method of any preceding claim, further comprising repeatedly making and reverting the change to obtain a plurality of first sub periods 402, 404, 406, 408, 410, 412) forming the first time period and a plurality of second sub periods (403, 405, 407, 409, 411 , 413) forming the second time period, wherein said adjusting at least one of the first and second time periods comprises filtering out at least one of the first or second sub periods.

8. The method of any preceding claim, wherein adjusting at least one of the first and second time periods comprises adjusting the first and second time periods such that distribution of values of the characterization indicator is substantially similar over the first adjusted time period and over the second adjusted time period.

9. The method of any preceding claim, wherein adjusting at least one of the first and second time periods comprises extending the second time period until the values of the characterization indicator observed during the second time period substantially cover the range of the values of the characterization indicator observed during the first time period; or vice versa, extending the first time period until the values of the characterization indicator observed during the first time period substantially cover the range of the values of the characterization indicator observed during the second time period.

10. The method of any preceding claim, wherein the change made in the communication network is related to one or more of power save, performance optimization, increasing capacity, load balancing and solving a performance problem.

11 . The method of any preceding claim, wherein the characterization indicator is substantially invariant to the change made in the communication network.

12. The method of any preceding claim, wherein the characterization indicator is related to load in the communication network.

13. The method of any preceding claim, wherein the performance indicator is spectral efficiency or user throughput and the characterization indicator is traffic volume or number of active users.

14. The method of any preceding claim, further comprising, responsive to the result of the evaluation of the change made in the communication network, keeping the change or reverting the change.

15. An apparatus (20, 111 , 112) comprising a processor (21 ), and a memory (22) including computer program code; the memory and the computer program code configured to, with the processor, cause the apparatus to perform the method of any one of claims 1 -14.

16. A computer program comprising computer executable program code (23) which when executed by a processor causes an apparatus to perform the method of any one of claims 1 -14.