WO2024004174A1

WO2024004174A1 - Automatic profile generation on test machine

Info

Publication number: WO2024004174A1
Application number: PCT/JP2022/026386
Authority: WO
Inventors: 仁中里; 光広朽津; 紗季田中; 遥堀内; 啓佑高見
Original assignee: 楽天モバイル株式会社
Priority date: 2022-06-30
Filing date: 2022-06-30
Publication date: 2024-01-04

Abstract

A test evaluation method according to the present disclosure includes: providing, on a virtual base, a first configuration for calculating actual data; setting a first measurement device for the first configuration; obtaining first data for the first configuration with the first measurement device; providing a second configuration in which one device in the first configuration has been replaced by a simulator; setting a second measurement device for the second configuration; obtaining second data for the second configuration with the second measurement device; and obtaining a differential profile for the first data and the second data.　The method may also include: evaluating the quality of the first configuration corresponding to a version of software for the first device, in accordance with the differential profile; including the quality evaluation of the first configuration in a release process ticket; and learning the relationship between the version and the number of bugs on the basis of the contents of the release process ticket.

Description

Automatic profile generation on test machine

The present disclosure relates to generating profiles for testing and evaluation of radio access networks on a virtualized infrastructure.

In the fifth generation mobile system (5G system), the virtualization of the core network (CN) is progressing, and following this, the virtualization of the radio access network (RAN) is attracting attention.
The functions of the RAN's Central Unit (CU) and Distributed Unit (DU) can be virtualized into virtualized CUs (vCUs) and virtualized DUs (vDUs), respectively.

Furthermore, due to the openness of O-RAN compliance, multi-vendoring is progressing, with multiple different vendors providing RAN CUs, DUs, and Radio Units (RUs).

On the other hand, for large-scale systems including multi-vendor systems, there is a method called ticket-driven development (TiDD). TiDD is a development style in which work is divided into tasks and managed by assigning them to tickets in a bug tracking system (BTS). TiDD follows the principle that all work and issues in software development are developed after filing a ticket.
BTS is a system for registering project bugs and tracking their modification status. More specifically, a ticket is issued as a bug registration, and the modification status of the project is tracked using the ticket. For example, Redmine (Non-Patent Document 1) is known as a tool for BTS.

RAN testing and evaluation is becoming more complex due to multi-vendoring, where RUs, DUs, and CUs are provided by multiple different vendors. Furthermore, due to multi-vendor technology, it is becoming more difficult to identify the location of quality deterioration during software updates.

The present disclosure is devised to solve at least one of the problems of the prior art, and provides a method for creating a profile for testing and evaluation of a radio access network on a virtualization infrastructure (test machine). do.

A management device according to the present disclosure configures a first measuring device for a first configuration on a virtualization platform, and obtains first data for the first configuration using the first measuring device. providing a second configuration in which one device in the first configuration is replaced with a simulator; setting a second measuring device for the second configuration; A processor is provided that executes the following steps: obtaining second data for the second configuration using a measuring instrument, and obtaining a difference profile between the first data and the second data.

The test evaluation system according to the present disclosure includes the management device and a ticket management system. The ticket management system includes a quality evaluation of the sent first configuration in a release process ticket, and uses the release process ticket to determine the relationship between the software version and the number of bugs for the one device. , and a processor to execute the .

A test evaluation method according to the present disclosure includes: providing a first configuration for calculating performance data on a virtualization platform; setting a first measuring device for the first configuration; obtaining first data for the first configuration using the first measuring instrument; providing a second configuration in which one device in the first configuration is replaced with a simulator; setting a second measuring device for the configuration; obtaining second data for the second configuration by the second measuring device; and a difference between the first data and the second data. and obtaining a profile.

FIG. 1 is a diagram illustrating an example of a wireless communication system to which a method or a management device according to the present disclosure is applied. FIG. 2 is a diagram illustrating an example of the operation of test evaluation by the management device according to the embodiment. FIG. 3 is a schematic diagram showing an example of the management device according to the embodiment. FIG. 4 is a diagram illustrating an example of application of the differential profile according to the embodiment. FIG. 5 is a sequence diagram showing an example of test evaluation according to the embodiment. FIG. 6 is a flowchart illustrating an example of the test evaluation method according to the embodiment.

Hereinafter, one embodiment of the present disclosure will be described in detail with reference to the drawings.

FIG. 1 shows an example of a wireless communication system to which a method or a management device according to the present disclosure is applied. The wireless communication system 1 in FIG. 1 includes a radio access network (RAN) 200 and a core network (CN) 500.
The RAN 200 includes a Central Unit (CU) 210, a Distributed Unit (DU) 220, and a Radio Unit (RU) 230, and implements the functions of a base station. In RAN 200, RU 230 can communicate with user equipment (UE) 300. Moreover, CU210 is connected to CN500.
The functions of the DU 220 and the CU 210 are virtualized on a virtualization infrastructure and are constructed as a virtualized DU (vDU) 220 and a virtualized CU (vCU) 210, respectively.
In the following description, unless there is a particular need, DU and vDU may be described without distinction, and CU and vCU may be described without distinction.

Furthermore, it is assumed that connections between devices (CU, DU, or RU elements) in the RAN 200 and connections with the CN 500 follow specifications standardized by the O-RAN Alliance (O-RAN) or the like. Note that the space between the RU 230 and the vDU 220 is called a fronthaul, the space between the vDU 220 and the vCU 210 is called a midhaul, and the space between the vCU 210 and the CN 500 is called a backhaul.
It is assumed that by standardizing connections between devices, it is possible for multiple vendors to provide the RU 230, vDU 220, and vCU 210 in the RAN 200 in FIG. 1 (multivendorization).
Furthermore, the management device 100 can connect to the vCU 210, vDU 220, and RU 230 and control them. The management device 100 may be an orchestrator (especially an E2E orchestrator).

Regarding the multi-vendor RAN 200 as shown in FIG. 1, testing and evaluation of the RAN is complicated because devices provided by different vendors are connected.
Furthermore, due to multi-vendor technology, it has become more difficult to identify the location where quality deterioration occurs when updating software in each virtualized device.

An example of the operation of the management device 100 for executing RAN test evaluation according to the embodiment will be described with reference to FIG. 2. First, the RAN 200 (see the upper part of FIG. 2), which is the target of test evaluation, will be described.
In the upper part of FIG. 2, the RAN 200 to be evaluated (hereinafter referred to as "first configuration" or "performance calculation configuration") includes a CU 210, a DU 220, and an RU 230 to be evaluated. Furthermore, a UE simulator 305 capable of transmitting and receiving test data is used as the UE. Further, as a CN, a CN simulator (core simulator) 505 capable of transmitting and receiving test data is used.

Further, for test evaluation, a measuring device 400 (also referred to as a "first measuring device") for collecting signals and the like within the RAN 200 is used. The measuring device 400 includes a radio wave capture 440, a fronthaul capture 430, a midhaul capture 420, and a backhaul capture 410. The radio wave capture 440 captures radio waves in order to analyze the quality of the radio waves from the RU 230 to the UE simulator 305. Fronthaul capture 430 collects signals between RU 230 and DU 220. Midhaul capture 420 collects signals between DU 220 and CU 210. Backhaul capture 410 collects signals between CU 210 and CN simulator 505.
The management device 100 analyzes or analyzes the signals or radio waves collected by the measuring device 400 for the RAN 200, and generates test result data 610 (also referred to as "first data"). The test result data for the performance calculation configuration is compared with the test result data for a configuration whose operation details have been confirmed in advance (referred to as the "second configuration" or "theoretical value calculation configuration").

The RAN 205 in the lower part of FIG. 2 is a configuration for calculating a theoretical value for test evaluation of the DU 220 (theoretical value calculation configuration).
In the RAN 205 in the lower part of FIG. 2, the DU 220 to be evaluated in the RAN 200 in the upper part of FIG. 2 is replaced with a DU simulator 225 whose operation contents have been confirmed in advance. Since the CU 210, RU 230, UE simulator 305, and CN simulator 505 are the same as the RAN 200, their explanations will be omitted.
Furthermore, the same measuring device 400 (referred to as a “second measuring device”) for collecting signals and the like for test evaluation in the RAN 205 can be used as in the RAN 200.

Regarding the RAN 205, the signals or radio waves collected by the measuring device 400 are analyzed or analyzed to obtain test result data 615 (also referred to as "second data").
The CU 210 and RU 230, the UE simulator 305, and the CN simulator 505 are common to the RAN 205 that is the theoretical value calculation configuration (second configuration) and the RAN 200 that is the actual performance calculation configuration (first configuration). One device (that is, DU 220) in the RAN 200 that has a performance calculation configuration is replaced with a simulator (DU simulator 225) whose operation has been confirmed in advance in the RAN 205 that has a theoretical value calculation configuration.
Therefore, by taking the difference between the test result data 610 for the RAN 200 and the test result data 615 for the RAN 205, the influence of the DU 220 can be calculated.

The management device 100 performs a test evaluation of the performance calculation configuration according to the test scenario 180 and creates test result data 610. The test scenario 180 may be an electronic file readable by the management device 100 that is prepared in advance to specify the content of the test evaluation. For example, the test scenario 180 may specify a device (DU in the example of FIG. 2) to be tested and evaluated. In particular, test scenario 180 may be provided by ticket management system 700.
The management device 100 configures a theoretical value calculation configuration by replacing the device to be evaluated in the performance calculation configuration with a simulator according to the test scenario 180, performs test evaluation, and creates test result data 615.

In the example of FIG. 2, the test scenario 180 specifies the DU 220 as the element to be evaluated. In this case, the DU 220 in the performance calculation configuration is replaced by the DU simulator 225.
In particular, in the performance calculation configuration, since the DU is virtualized on the virtualization platform, the replacement of the DU 220 by the DU simulator 225 can be automatically performed by software without any human intervention.
The management device 100 uses the measuring instrument 400 to collect data related to the operation of each device in the performance calculation configuration and the theoretical value calculation configuration, analyzes or analyzes the data, and stores the data as test result data 610 and test result data 615, respectively.

The management device 100 further creates a difference profile 650 from the difference between the test result data 610 and the test result data 615. The management device 100 determines the normality of the operation caused by incorporating the DU 220 into the RAN 200 based on the difference profile. This determination result is called quality evaluation for the RAN 200. The quality evaluation is sent from the management device 100 to the ticket management system 700.
Thereby, the ticket management system 700 can write the quality evaluation for the RAN 200 in the ticket. Tickets will be discussed later.

According to the management device 100, a difference profile can be automatically obtained based on the given test scenario 180. In other words, the actual calculation configuration and the theoretical value calculation configuration are provided on the virtualization platform (test machine), and the human intervention is required until the test result data is obtained by setting the measuring instruments for each of the actual value calculation configuration and the theoretical value calculation configuration. I have nothing to do.

FIG. 3 is a schematic diagram illustrating a configuration example of the management device 100 for performing RAN test evaluation according to the embodiment. Management device 100 includes a transmitting/receiving section 110 and a processing section 120. The management device 100 may further include a configuration not shown in FIG. 3.
The transmitting/receiving unit 110 transmits and receives data to and from the CU 210, DU 220, RU 230, DU simulator 225, measuring instrument 400, and ticket management system (bug tracking system) 700 in FIG. The transmitting/receiving unit 110 may also be configured to transmit and receive data with the UE simulator 305 or the CN simulator 505.

The processing unit 120 includes a processor 122 and a memory 124. Note that the number of processor 122 and memory 124 may be one or more. The processing unit 120 may further include storage 126. The processing unit 120 operates the transmitting/receiving unit 110, and can perform data processing as the management device 100 using the processor 122 and memory 124. A test scenario 180 may be stored in the storage 126.

The processor 122 of the processing unit 120 can execute operations in the management device 100 in order to perform the RAN test evaluation described with reference to FIG. 2.

FIG. 4 shows an example of application of the difference (difference profile 650) of test result data (see

reference numerals

610 and 615 in FIG. 2) to the actual value calculation configuration and the theoretical value calculation configuration.
In FIG. 4, the difference profile 650 is associated with a bug tracking system (BTS) ticket 660. In particular, ticket 660 may include a quality assessment of a new release of DU 220. Alternatively, the ticket 660 may be a release process ticket that includes a quality evaluation regarding the software version upgrade related to the DU 220. New release tickets and release process tickets can be used in ticket-driven development (TiDD).

FIG. 4 shows an example of the contents written on the ticket.
Information regarding automatic generation (Production Automation) of the RAN is written in the ticket. The information may include some of the following examples.
(1) RAN deployment configuration. As an example, "1vCU/4DU/3ORU" indicates that the RAN consists of one vCU, four DUs, and three ORUs (RUs with O-RAN specifications).
(2) RAN threshold. For example, it relates to information on the beam ID or the number of MIMO (Multi Input Multi Output) layers.
(3) O-RAN Certification Limited. For example, it relates to restrictions such as only supporting IPv6.
(4) Type of virtualization COTS (Commercial Off The Shelf). For example, information regarding FPGA support.

The ticket contains information about the monitoring system. Information about the monitoring system may include definitions of monitoring conditions. The definition of the monitoring condition includes, for example, a threshold value indicating a permissible value for the measurement data, a monitoring number indicating the number of data items to be monitored, or information regarding monitoring items.

FIG. 5 is a sequence diagram for test evaluation when upgrading the DU software by the test evaluation system including the management device 100 and the ticket management system 700 according to the embodiment. A ticket management system (bug tracking system) 700 allocates tickets to version upgrades of DUs and manages bug handling and the like.
The ticket management system 700 issues a ticket and sends the generated test scenario (see reference numeral 180 in FIG. 2) to the management device 100 when new software for DU is released.

The management device 100 instructs the virtualization infrastructure to generate a performance calculation configuration (simply referred to as "configuration A") based on the test scenario. As shown in FIG. 2, configuration A includes a RAN 200 having a CU 210, an upgraded DU 220, and an RU 230. Also, a measuring instrument 400 (simply referred to as "measuring instrument #A") is set for the performance calculation configuration.
Measuring device #A acquires measurement data in configuration A and sends it to management device 100. The management device 100 creates test result data (see reference numeral 610 in FIG. 2) from the data.

The management device 100 instructs the virtualization infrastructure to generate a theoretical value calculation configuration (simply referred to as "configuration B") based on the test scenario. Configuration B includes a RAN 205 with a CU 210, a DU simulator 225, and an RU 230, as shown in FIG. Furthermore, the management device 100 sets the measuring device 400 (simply referred to as “measuring device #B”) for the theoretical value calculation configuration.
As described above, since the DU is virtualized on the virtualization platform, the DU of configuration A can be automatically replaced by the DU simulator of configuration B using software.
Measuring device #B acquires data in configuration B and sends it to management device 100. The management device 100 creates test result data (see reference numeral 615 in FIG. 2) from the data.
Regarding the sequence diagram in FIG. 5, the explanation was given in the order of generation of configuration A, setting and data acquisition of measuring instrument #A, generation of configuration B, setting of measuring instrument #B and data acquisition, but these steps are explained below. The order of execution is not limited to this. For example, generation of configuration B, setting of measuring instrument #B, and data acquisition may be performed before generation of configuration A, setting of measuring instrument #A, and data acquisition.

The management device 100 further creates a difference profile (see 650 in FIG. 2) from the test result data for configuration A (see 610 in FIG. 2) and the test result data for configuration B (see 615 in FIG. 2). The management device 100 determines the normality of the operation associated with the version upgrade of the DU based on the difference profile, and sends the determination result (quality evaluation) to the ticket management system 700. The ticket management system 700 writes the sent determination result in the ticket.

The ticket management system 700 may learn the relationship between the software version of the DU and the number of bugs based on the contents of the ticket.
The ticket management system 700 includes the following steps: including the submitted quality evaluation in the release process ticket; and learning the relationship between the software version and the number of bugs for one device using the release process ticket. It may include one or more processors for execution.

A test evaluation method 1000 according to the embodiment will be described with reference to FIG. 6. This method includes the following steps.
A first configuration for calculating performance data is provided (1010). An example of the first configuration is the performance calculation configuration shown in FIG.
A first meter for a first configuration is configured (1020). An example of the first measuring device is the measuring device 400 for the performance calculation configuration shown in FIG.
A first measurement device obtains first data for a first configuration (1030). An example of the first data is test result data 610 in FIG.

A second configuration is provided in which one device in the first configuration is replaced with a simulator (1040). An example of the second configuration is a theoretical value calculation configuration in which the DU 220 in the actual performance calculation configuration in FIG. 2 is replaced with the DU simulator 225.
A second meter for a second configuration is configured (1050). An example of the second measuring device is the measuring device 400 for the theoretical value calculation configuration shown in FIG.
A second measurement device obtains second data for a second configuration (1060). An example of the second data is the test result data 615 in FIG.

A differential profile between the first data and the second data is obtained (1070). An example of a differential profile is differential profile 650 in FIG. 2 .
Note that the above steps (1010 to 1070) are not limited to execution in the above order as long as a difference profile between the first data and the second data can be obtained. For example, after obtaining the second data using the second configuration and the second measuring device, the first data may be obtained using the first configuration and the first measuring device.

Note that the test evaluation method 1000 may further include the following steps.
Quality evaluation of a first configuration corresponding to a software version for one device is performed according to the difference profile (1080). As an example of the quality evaluation of the first configuration, among the number of monitoring items and monitoring items, items (bugs) that are not recognized as normal operation according to the threshold value that gives the permissible value of the measured data value in the monitoring conditions in Figure 4. can be the number of Further, an example of a software version for one device is a DU software version.
A quality evaluation of the first configuration is included in the release process ticket (1090). For example, the number of bugs for each release version may be written on a ticket in a bug tracking system (BTS).

Based on the contents of the release process ticket, the relationship between the version and the number of bugs is learned (1100). For example, the number of bugs for each release version written in a ticket may be aggregated, and the relationship between the release version and the number of bugs may be learned by machine.

Furthermore, the present disclosure also includes a program for causing a system to execute the above-described test evaluation method 1000. The program may be provided recorded on a computer-readable non-transitory storage medium.

As described above, according to the present disclosure, the performance calculation configuration and the theoretical value calculation configuration are provided on the virtualization platform, and the test result data is obtained by setting measuring instruments for each of the performance calculation configuration and the theoretical value calculation configuration. There is no need for human intervention.

The present disclosure is not limited to the above-described embodiments, but also includes various modifications in which components are added, deleted, or converted to the above-described configuration. Moreover, each embodiment can be combined in various ways.

Note that the term "connection" used in this description means a logical connection for communication. For example, "RU connected to vDU" means that the vDU and RU are logically connected so that they can communicate. The vDU and RU do not necessarily have to be physically directly connected by a physical cable or the like, and a plurality of devices or wireless communication may be interposed between the vDU and RU.

Further, the present disclosure includes the following aspects.

[1] A management device,
configuring a first measurement device for a first configuration on a virtualization infrastructure;
obtaining first data for the first configuration by the first measuring device;
providing a second configuration in which one device in the first configuration is replaced with a simulator;
configuring a second measuring device for the second configuration;
obtaining second data for the second configuration by the second measuring device;
obtaining a differential profile between the first data and the second data;
A management device comprising a processor that executes.

[2] The processor,
Evaluating the quality of the first configuration according to the difference profile;
Sending a quality evaluation of the first configuration to a ticket management system;
The management device according to [1], which further executes.

[3] The management device according to [2], wherein the quality evaluation of the first configuration corresponds to a software version for the one device.

[4] A management device,
configuring a first measurement device for a first configuration on a virtualization infrastructure;
obtaining first data for the first configuration by the first measuring device;
providing a second configuration in which one device in the first configuration is replaced with a simulator;
configuring a second measuring device for the second configuration;
obtaining second data for the second configuration by the second measuring device;
obtaining a differential profile between the first data and the second data;
Evaluating the quality of the first configuration corresponding to a software version for the one device according to the difference profile;
Sending a quality evaluation of the first configuration to a ticket management system;
a management device comprising a processor that executes;
A ticket management system,
including a quality evaluation of the sent first configuration in a release process ticket;
learning a relationship between a software version and a number of bugs for the one device using the release process ticket;
a ticket management system; and a ticket management system.

[5] Providing a first configuration for calculating performance data on a virtualization platform;
configuring a first measuring device for the first configuration;
obtaining first data for the first configuration by the first measuring device;
providing a second configuration in which one device in the first configuration is replaced with a simulator;
configuring a second measuring device for the second configuration;
obtaining second data for the second configuration by the second measuring device;
obtaining a differential profile between the first data and the second data;
test evaluation methods including;

[6] Evaluating the quality of the first configuration corresponding to a software version for the one device according to the difference profile;
including a quality evaluation of the first configuration in a release process ticket;
The test evaluation method according to [5], further comprising learning the relationship between the version and the number of bugs based on the contents of the release process ticket.

1 Wireless communication system 100 Management device 110 Transmission/reception unit 120 Processing unit 122 Processor 124 Memory 126 Storage 180

Test scenario

200, 205 Radio access network (RAN)
210 cu.
220 D.U.
225 DU Simulator 230 RU
300 User terminal (UE)
305 UE simulator 400 Measuring device 410 Backhaul capture 420 Midhaul capture 430 Fronthaul capture 440 Radio wave capture 500 Core network (CN)
505

CN simulator

610, 615 Test result data 650 Difference profile 660 Ticket 700 Ticket management system 1000 Test evaluation method

Claims

A management device,
configuring a first measurement device for a first configuration on a virtualization infrastructure;
obtaining first data for the first configuration by the first measuring device;
providing a second configuration in which one device in the first configuration is replaced with a simulator;
configuring a second measuring device for the second configuration;
obtaining second data for the second configuration by the second measuring device;
obtaining a differential profile between the first data and the second data;
A management device comprising a processor that executes.
The processor,
Evaluating the quality of the first configuration according to the difference profile;
Sending a quality evaluation of the first configuration to a ticket management system;
The management device according to claim 1, further performing:
The management device according to claim 2, wherein the quality evaluation of the first configuration corresponds to a software version for the one device.
A management device,
configuring a first measurement device for a first configuration on a virtualization infrastructure;
obtaining first data for the first configuration by the first measuring device;
providing a second configuration in which one device in the first configuration is replaced with a simulator;
configuring a second measuring device for the second configuration;
obtaining second data for the second configuration by the second measuring device;
obtaining a differential profile between the first data and the second data;
Evaluating the quality of the first configuration corresponding to a software version for the one device according to the difference profile;
Sending a quality evaluation of the first configuration to a ticket management system;
a management device comprising a processor that executes;
A ticket management system,
including a quality evaluation of the sent first configuration in a release process ticket;
learning a relationship between a software version and a number of bugs for the one device using the release process ticket;
a ticket management system; and a ticket management system.
Providing a first configuration for calculating performance data on a virtualization platform;
configuring a first measuring device for the first configuration;
obtaining first data for the first configuration by the first measuring device;
providing a second configuration in which one device in the first configuration is replaced with a simulator;
configuring a second measuring device for the second configuration;
obtaining second data for the second configuration by the second measuring device;
obtaining a differential profile between the first data and the second data;
Test evaluation methods including.
evaluating the quality of the first configuration corresponding to a software version for the one device according to the difference profile;
including a quality evaluation of the first configuration in a release process ticket;
The test evaluation method according to claim 5, further comprising: learning a relationship between the version and the number of bugs based on the contents of the release process ticket.