WO2022061784A1 - 通信方法、装置及系统 - Google Patents

通信方法、装置及系统 Download PDF

Info

Publication number
WO2022061784A1
WO2022061784A1 PCT/CN2020/117940 CN2020117940W WO2022061784A1 WO 2022061784 A1 WO2022061784 A1 WO 2022061784A1 CN 2020117940 W CN2020117940 W CN 2020117940W WO 2022061784 A1 WO2022061784 A1 WO 2022061784A1
Authority
WO
WIPO (PCT)
Prior art keywords
model
information
nwdaf
network element
performance
Prior art date
Application number
PCT/CN2020/117940
Other languages
English (en)
French (fr)
Inventor
黄谢田
秦东润
王楚捷
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to PCT/CN2020/117940 priority Critical patent/WO2022061784A1/zh
Priority to CA3193840A priority patent/CA3193840A1/en
Priority to AU2021347699A priority patent/AU2021347699A1/en
Priority to PCT/CN2021/085428 priority patent/WO2022062362A1/zh
Priority to CN202180041882.4A priority patent/CN115699848A/zh
Priority to EP21870751.1A priority patent/EP4207860A4/en
Publication of WO2022061784A1 publication Critical patent/WO2022061784A1/zh
Priority to US18/188,205 priority patent/US20230224752A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/70—Admission control; Resource allocation
    • H04L47/83—Admission control; Resource allocation based on usage prediction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W24/00—Supervisory, monitoring or testing arrangements
    • H04W24/10—Scheduling measurement reports ; Arrangements for measurement reports
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/70—Admission control; Resource allocation
    • H04L47/82—Miscellaneous aspects
    • H04L47/822—Collecting or measuring resource availability data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W24/00—Supervisory, monitoring or testing arrangements
    • H04W24/04—Arrangements for maintaining operational condition

Definitions

  • the present application relates to the field of communication technologies, and in particular, to a communication method, device, and system.
  • the training of the machine learning model is usually carried out by learning the mapping between a certain set of input features and the output target, and by optimizing some loss functions, the output results of the machine learning model (ie, predicted values) can be compared with the actual results (ie, labels). value/true value) to minimize the error.
  • the optimal model is trained, the output of the model is used to predict the future situation.
  • the data that will be used in the future is similar to the data used during model training.
  • the distribution of input features at training time and input features at prediction time remains constant.
  • this assumption is usually not established.
  • the characteristics of the data will change over time due to changes in network deployment, changes in application layer business requirements, and changes in the distribution of actual network users. Therefore, the performance of the model (that is, the generalization ability) will change. gradually decreased over time.
  • the specific performance may be that the accuracy of the model decreases, that is, the error between the predicted value of the model and the true value becomes larger.
  • the training data analysis network element cannot perceive the data analysis network element that supports the inference function (referred to as the inference data analysis network element). ), and the inferred data analysis network element is not capable of model training. Therefore, when the model performance is degraded, if it is inferred that the data analysis network element continues to use the degraded model for data analysis, the data analysis result will be inaccurate.
  • the present application provides a communication method, device, and system to implement retraining of the model in time when the performance of the model is degraded, so as to ensure the performance of the model.
  • an embodiment of the present application provides a communication method, including: a first data analysis network element receiving first information from a second data analysis network element, where the first information includes a performance report of a model, The performance report is used to indicate the evaluation result of the performance of the model, or the performance report of the model is used to indicate that the evaluation result of the performance of the model does not meet the requirements of the performance index of the model; the first data analysis The network element updates the first model information of the model according to the performance report of the model, and obtains the second model information of the model; the first data analysis network element sends the second information to the second data analysis network element , the second information includes the second model information.
  • the second data analysis network element can send a performance report of the model to the first data analysis network element, so that the first data analysis network element can
  • the performance report updates the model, obtains the second model information of the model, and sends the second model information to the second data analysis network element, so that the second data analysis network element can update the model based on the second model information, so that the
  • the model is trained in time to ensure the performance of the model.
  • the first data analysis network element sends third information to the second data analysis network element, where the third information includes the performance index of the model, the performance index of the model Used to obtain evaluation results of the performance of the model.
  • the first data analysis network element may send the performance index of the model to the second data analysis network element in advance, so that the second data analysis network element generates a performance report of the model based on the performance index of the model, which is helpful for the second data analysis network element.
  • a data analysis network element determines whether to start model training, and improves model performance after model training.
  • the first data analysis network element sends the second information to a third data analysis network element.
  • the first data analysis network element can not only send the second information to the second data analysis network element, but also can send the second information to other network elements using the model, such as the third data analysis network element, so that the first data analysis network element
  • the three data analysis network element can also use the second model information to update the model to improve the use effect of the model.
  • the first data analysis network element receiving the first information from the second data analysis network element includes: the first data analysis network element receives information from the second data analysis network element through a network storage network element 2.
  • Sending the second information by the first data analysis network element to the second data analysis network element includes: the first data analysis network element sending the second data analysis network element to the second data analysis network element through a network storage network element. 2. Information.
  • the network storage network element can be used as an intermediate network element to realize the model update interaction between the first data analysis network element and the second data analysis network element, which can be applied to the first data analysis network element and the second data analysis network element. Analyze scenarios where there are no interfaces between NEs.
  • an embodiment of the present application provides a communication method, including: a second data analysis network element sending first information to a first data analysis network element, where the first information includes a performance report of a model, the performance of the model The report is used to indicate the evaluation result of the performance of the model, or the performance report of the model is used to indicate that the evaluation result of the performance of the model does not meet the requirements of the performance index of the model; the second data analysis network element receives second information from the first data analysis network element, the second information includes second model information of the model, the second model information is to update the model according to the performance report of the model The first model information is obtained; the second data analysis network element updates the model according to the second model information.
  • the second data analysis network element can send a performance report of the model to the first data analysis network element, so that the first data analysis network element can
  • the performance report updates the model, obtains the second model information of the model, and sends the second model information to the second data analysis network element, so that the second data analysis network element can update the model based on the second model information, so that the
  • the model can be retrained in time to ensure the performance of the model.
  • the second data analysis network element receives third information from the first data analysis network element, where the third information includes the performance index of the model, the performance of the model Metrics are used to obtain evaluation results of the performance of the model.
  • the first data analysis network element can send the performance index of the model to the second data analysis network element in advance, so that the second data analysis network element can generate a performance report of the model based on the performance index of the model, which is helpful for the second data analysis network element.
  • a data analysis network element determines whether to start model training, and improves the accuracy of model training.
  • sending the first information by the second data analysis network element to the first data analysis network element includes: the second data analysis network element sends the first data to the first data analysis network element through a network storage network element The analysis network element sends the first information; and the second data analysis network element receives the second information from the first data analysis network element, including: the second data analysis network element receives data from the network storage network element. The first data analyzes the second information of the network element.
  • the network storage network element can be used as an intermediate network element to realize the model update interaction between the first data analysis network element and the second data analysis network element, which can be applied to the first data analysis network element and the second data analysis network element. Analyze scenarios where there are no interfaces between NEs.
  • the model performance indicators include one or more of the following: precision rate, accuracy rate, error rate, recall rate, F1 score, mean square error, root mean square error, root mean square pair Number error, mean absolute error, model inference time, model robustness, model scalability, model interpretability.
  • the third information further includes one or more of the following: an analysis type identifier, an identifier of the model, and an identifier of a sub-model, where the analysis type identifier is used to indicate the Analysis type.
  • the third information further includes one or more of the following: a reporting period and threshold information, where the reporting period is used to indicate the time for reporting the performance report of the model, and the threshold information Used to indicate the conditions under which the performance report of the model is reported.
  • the first data analysis network element may instruct the second data analysis network element to report the time and/or condition of the performance report of the model, thereby implementing conditional reporting and saving resource overhead.
  • the first information further includes one or more of the following information corresponding to the performance report of the model: time, region, slice.
  • the model performance after the retraining of the model is performed by the first data analysis network element can be improved.
  • the second information further includes one or more of the following: the identification of the model, the identification of the sub-model, the performance evaluation result of the model, the corresponding performance evaluation result of the model hardware capability information, the size of the model, and the inference duration of the model.
  • the first data analysis network element sends one or more of the performance evaluation result of the model, the hardware capability information corresponding to the performance evaluation result of the model, the size of the model, or the inference duration of the model to the second data analysis network element , which is helpful for the second data analysis network element to determine whether to use the model, thereby reducing the waste of resource overhead.
  • an embodiment of the present application provides a communication method, including: a first data analysis network element updates first information of a model to second information of the model; the first data analysis network element determines the model The index information of the second information, the index information of the second information includes first identification information, and the first identification information is used to indicate the second information of the model; the first data analysis network element reports to the The second data analysis network element sends the index information of the second information, and the index information of the second information is used for acquiring the second information of the model.
  • the index information of the second information of the model may also be referred to as model index information corresponding to the second information.
  • the first data analysis network element updates the model, and after obtaining the second information of the model, the index information of the second information can be sent to the second data analysis network element, so that the second data analysis network element can
  • the index information acquires new model information, that is, the second information, and then the second data analysis network element can update the model according to the new model information, so as to improve the performance of the model.
  • the index information of the second information further includes one or more of the following: an analysis type identifier corresponding to the model, an identifier of the model, and a version of the second information of the model information.
  • the first data analysis network element receives index information of the first information of the model from the second data analysis network element, and the index information of the first information includes a second identifier information, the second identification information is used to indicate the first information of the model; the first data analysis network element obtains the first information of the model according to the index information of the first information.
  • the index information of the first information further includes one or more of the following: an analysis type identifier corresponding to the model, an identifier of the model, and a version of the first information of the model information.
  • the first data analysis network element updates the first information of the model to the second information of the model, including: the first data analysis network element analyzes the network element from the second data Obtain a first request, where the first request is used to update the first information of the model, and the first request includes index information of the first information of the model; the first data analysis network element according to the first information The index information of a piece of information obtains the first information of the model; the first data analysis network element updates the first information of the model to obtain the second information of the model.
  • the first data analysis network element receives the index information of the first information of the model from the second data analysis network element, including: the first data analysis network element reports to the second data analysis network element element sends a second request, the second request is used to request the index information of the first information of the model, the second request includes the analysis type identifier corresponding to the model; the first data analysis network element is obtained from the The second data analysis network element receives a second response, where the second response includes index information of the first information of the model.
  • the first data analysis network element receives index information of the first information of the model from the second data analysis network element through a network storage network element.
  • the first data analysis network element sends the index information of the second information of the model to the second data analysis network element through a network storage network element.
  • the first data analysis network element is a client-side data analysis network element in distributed learning
  • the second data analysis network element is a server-side data analysis network element in distributed learning .
  • the distributed learning is federated learning.
  • the first data analysis network element is a data analysis network element supporting an inference function
  • the second data analysis network element is a data analysis network element supporting a training function
  • an embodiment of the present application provides a communication device, where the device may be a data analysis network element or a chip used for the data analysis network element.
  • the device has the function of implementing the first aspect to the third aspect, or each possible implementation method of the first aspect to the third aspect. This function can be implemented by hardware or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • an embodiment of the present application provides a communication device, including a processor and a memory; the memory is used to store computer-executed instructions, and when the device is running, the processor executes the computer-executed instructions stored in the memory, so that the The apparatus executes any of the above-mentioned methods of the first to third aspects and possible implementation methods of the first to third aspects.
  • an embodiment of the present application provides a communication device, including a communication device for performing each step of the methods of the first aspect to the third aspect and any of the possible implementation methods of the first aspect to the third aspect. Units or means.
  • an embodiment of the present application provides a communication device, including a processor and an interface circuit, where the processor is configured to communicate with other devices through the interface circuit, and execute the methods of the first to third aspects and the first aspect Any of the possible implementation methods to the third aspect.
  • the processor includes one or more.
  • an embodiment of the present application provides a communication device, including a processor that is connected to a memory and used to call a program stored in the memory to execute the methods of the first to third aspects and the first Any of the possible implementation methods of the aspects to the third aspect.
  • the memory may be located within the device or external to the device.
  • the processor includes one or more.
  • an embodiment of the present application further provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium runs on a computer, the processor causes the processor to execute the above-mentioned first to third aspects The method and any of the possible implementation methods of the first aspect to the third aspect.
  • an embodiment of the present application further provides a computer program product, the computer product includes a computer program, when the computer program runs, the above-mentioned methods of the first to third aspects and each of the first to third aspects are enabled. Any of the possible implementations.
  • an embodiment of the present application further provides a chip system, including: a processor configured to execute the methods of the first aspect to the third aspect and any of the possible implementation methods of the first aspect to the third aspect any method.
  • an embodiment of the present application further provides a communication system, including: a first data analysis network element configured to execute the foregoing first aspect or any implementation method of the first aspect, and a first data analysis network element configured to execute the foregoing second aspect or The second data analysis network element of any implementation method of the second aspect.
  • Figure 1 is a schematic diagram of the 5G network architecture
  • Figure 2 is a schematic diagram of the NF registration/discovery/update process in a 5G network
  • Figure 3 is a schematic diagram of the workflow of training NWDAF and inferring NWDAF under a separate architecture for training and inference;
  • FIG. 4 is a schematic diagram of a network architecture to which an embodiment of the present application is applicable.
  • 5 to 12 are schematic diagrams of eight methods for ensuring model validity in a training-inference separation scenario provided by an embodiment of the present application
  • FIG. 13 is a schematic diagram of a communication method provided by an embodiment of the present application.
  • Figure 14(a) shows the training process of horizontal federated learning
  • FIG. 14(b) is a schematic diagram of yet another communication method provided by an embodiment of the present application.
  • FIG. 15 is a schematic diagram of a communication device according to an embodiment of the present application.
  • FIG. 16 is a schematic diagram of still another communication apparatus provided by an embodiment of the present application.
  • the wireless machine learning model-driven network architecture (wireless Machine Learning-based Network, wMLN) mainly solves the problem of life cycle management of machine learning models in wireless networks.
  • the model training function and model inference function in this network architecture are two core functional modules closely related to the machine learning model.
  • the model training function requires high computing power and requires a large amount of data. It usually needs to be deployed on a centralized network element with powerful computing power and data.
  • the model inference function is usually deployed in the local network element close to the service function to reduce the transmission and processing delay. Therefore, the separation of model training functions and inference functions is a typical deployment scenario.
  • NWDAF Network Data Analytics Function
  • 3GPP 3rd generation partnership project
  • NWDAF can utilize machine learning models for data analysis.
  • the functions of NWDAF in 3GPP Release 17 are decomposed, including data collection functions, model training functions, and model inference functions.
  • the NWDAF that deploys the training function (referred to as the training NWDAF) can provide the trained model and the NWDAF that deploys the inference function. (referred to as Inference NWDAF) by obtaining the model provided by the training NWDAF to perform model inference and provide data analysis services.
  • the training of the machine learning model is usually carried out by learning the mapping between a certain set of input features and the output target, and by optimizing some loss functions, the output results of the machine learning model (ie, predicted values) can be compared with the actual results (ie, labels). value/true value) to minimize the error.
  • the optimal model is trained, the output of the model is used to predict the future situation.
  • the data that will be used in the future is similar to the data used during model training.
  • the distribution of input features at training time and input features at prediction time remains constant.
  • this assumption is usually not established.
  • the characteristics of the data will change over time due to changes in network deployment, changes in application layer business requirements, and changes in the distribution of actual network users. Therefore, the performance of the model (that is, the generalization ability) will change. gradually decreased over time.
  • the specific performance may be that the accuracy of the model decreases, that is, the error between the predicted value of the model and the true value becomes larger.
  • the 5G network architecture shown in Figure 1 can include three parts, namely the terminal equipment part, the data network (DN) and the operator network part. The following briefly describes the functions of some of the network elements.
  • the operator network may include one or more of the following network elements: Authentication Server Function (AUSF) network element, Network Exposure Function (NEF) network element, Policy Control Function (Policy Control Function) Function, PCF) network element, unified data management (unified data management, UDM), unified database (Unified Data Repository, UDR), network storage function (Network Repository Function, NRF) network element, application function (Application Function, AF) network element, Access and Mobility Management Function (AMF) network element, session management function (SMF) network element, RAN and user plane function (UPF) network element, NWDAF network element, etc.
  • AUSF Authentication Server Function
  • NEF Network Exposure Function
  • Policy Control Function Policy Control Function
  • PCF Policy Control Function
  • UDM Unified Data Repository
  • NRF Network Repository Function
  • AMF Access and Mobility Management Function
  • SMF session management function
  • UPF user plane function
  • the terminal device in this embodiment of the present application may be a device for implementing a wireless communication function.
  • the terminal equipment may be a user equipment (UE), an access terminal, a terminal unit, a terminal station, a mobile station, a mobile station in a 5G network or a public land mobile network (PLMN) evolved in the future.
  • UE user equipment
  • PLMN public land mobile network
  • remote station remote terminal
  • mobile device wireless communication device
  • terminal agent or terminal device etc.
  • the access terminal may be a cellular phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a wireless communication Functional handheld devices, computing devices or other processing devices connected to wireless modems, in-vehicle devices or wearable devices, virtual reality (VR) end devices, augmented reality (AR) end devices, industrial control (industrial) wireless terminal in control), wireless terminal in self-driving, wireless terminal in remote medical, wireless terminal in smart grid, wireless terminal in transportation safety Terminals, wireless terminals in smart cities, wireless terminals in smart homes, etc. Terminals can be mobile or stationary.
  • SIP session initiation protocol
  • WLL wireless local loop
  • PDA personal digital assistant
  • a wireless communication Functional handheld devices computing devices or other processing devices connected to wireless modems, in-vehicle devices or wearable devices, virtual reality (VR) end devices, augmented reality (AR) end devices, industrial control (industrial) wireless terminal in control), wireless terminal in self-driving,
  • the above-mentioned terminal device can establish a connection with the operator network through an interface (eg, N1, etc.) provided by the operator network, and use the data and/or voice services provided by the operator network.
  • the terminal device can also access the DN through the operator's network, and use the operator's service deployed on the DN and/or the service provided by a third party.
  • the above-mentioned third party may be a service party other than the operator's network and the terminal device, and may provide other data and/or voice services for the terminal device.
  • the specific expression form of the above third party can be specifically determined according to the actual application scenario, and is not limited here.
  • RAN is a sub-network of an operator's network, and is an implementation system between service nodes and terminal equipment in the operator's network.
  • the terminal device To access the operator's network, the terminal device first passes through the RAN, and then can be connected to the service node of the operator's network through the RAN.
  • the RAN device in this application is a device that provides a wireless communication function for a terminal device, and the RAN device is also called an access network device.
  • the RAN equipment in this application includes but is not limited to: next-generation base station (g nodeB, gNB), evolved node B (evolved node B, eNB), radio network controller (radio network controller, RNC), node B in 5G (node B, NB), base station controller (BSC), base transceiver station (base transceiver station, BTS), home base station (for example, home evolved nodeB, or home node B, HNB), baseband unit (baseBand unit, BBU), transmission point (transmitting and receiving point, TRP), transmitting point (transmitting point, TP), mobile switching center, etc.
  • next-generation base station g nodeB, gNB
  • evolved node B evolved node B
  • eNB evolved node B
  • RNC radio network controller
  • node B in 5G node B, NB
  • base station controller BSC
  • base transceiver station base transceiver station
  • BTS home base station
  • base station for example, home
  • the AMF network element mainly performs functions such as mobility management and access authentication/authorization. In addition, it is also responsible for transferring user policies between UE and PCF.
  • the SMF network element mainly performs functions such as session management, execution of control policies issued by PCF, selection of UPF, and allocation of UE Internet Protocol (IP) addresses.
  • IP Internet Protocol
  • the UPF network element as the interface UPF with the data network, implements functions such as user plane data forwarding, session/flow-level accounting statistics, and bandwidth limitation.
  • the UDM network element is mainly responsible for the management of contract data, user access authorization and other functions.
  • UDR is mainly responsible for the access function of contract data, policy data, application data and other types of data.
  • the NEF network element is mainly used to support the opening of capabilities and events.
  • the AF network element mainly conveys the requirements of the application side to the network side, such as quality of service (Quality of Service, QoS) requirements or user status event subscriptions.
  • the AF may be a third-party functional entity or an application service deployed by an operator, such as an IP Multimedia Subsystem (IP Multimedia Subsystem, IMS) voice call service.
  • IP Multimedia Subsystem IP Multimedia Subsystem
  • the PCF network element is mainly responsible for policy control functions such as charging for sessions and service flow levels, QoS bandwidth guarantee and mobility management, and UE policy decision-making.
  • the NRF network element can be used to provide the network element discovery function, and provide network element information corresponding to the network element type based on the request of other network elements.
  • NRF also provides network element management services, such as network element registration, update, de-registration, and network element status subscription and push.
  • AUSF network element It is mainly responsible for authenticating users to determine whether to allow users or devices to access the network.
  • a DN is a network outside the operator's network.
  • the operator's network can access multiple DNs, and multiple services can be deployed on the DNs, which can provide data and/or voice services for terminal devices.
  • DN is the private network of a smart factory.
  • the sensors installed in the workshop of the smart factory can be terminal devices, and the control server of the sensor is deployed in the DN, and the control server can provide services for the sensor.
  • the sensor can communicate with the control server, obtain the instruction of the control server, and transmit the collected sensor data to the control server according to the instruction.
  • the DN is an internal office network of a company.
  • the mobile phones or computers of employees of the company can be terminal devices, and the mobile phones or computers of employees can access information and data resources on the internal office network of the company.
  • Nnwdaf, Nausf, Nnef, Npcf, Nudm, Naf, Namf, Nsmf, N1, N2, N3, N4, and N6 are interface serial numbers.
  • interface serial numbers refer to the meanings defined in the 3GPP standard protocol, which is not limited here.
  • the data analysis network element may be the NWDAF network element shown in FIG. 1 , or may be other network elements having the functions of the above NWDAF network element in the future communication system.
  • the network storage network element may be the NRF network element shown in FIG. 1 , or may be other network elements having the functions of the above NRF network element in the future communication system.
  • the data analysis network element is an NWDAF network element
  • the network storage network element is an NRF network element as an example for description.
  • NWDAF network elements are further divided into training NWDAF network elements and inference NWDAF network elements.
  • NRF network function
  • the network function here can be SMF, AMF, NEF, AUSF, NWDAF, PCF and so on.
  • NRF Network Function
  • NF registration/update/de-registration The available NF instance (NF instance) registers the services it can provide in the NRF, and the registration information is described by the NF profile (NF profile).
  • the NF profile includes NF types and NF services. Name, NF address and other information.
  • NRF is responsible for maintaining these NF configuration files. When the NF needs to be updated or deleted, the NRF will modify and delete the NF configuration file accordingly.
  • NF discovery The NRF receives the NF discovery request from the NF instance, and provides the discovered NF instance information to the requesting NF instance. For example, AMF requests NRF to discover SMF instances. For another example, an AMF requests the NRF to discover another AMF instance.
  • NF status notification The NRF notifies the subscribed NF service consumers of newly registered/updated/deregistered NF instances and the NF services provided by them.
  • the NF registration process includes steps 201 to 203 .
  • Step 201 NF1 sends an NF registration request to the NRF, carrying the NF configuration file.
  • the NF configuration file includes information such as NF type, NF service name, and NF address.
  • Step 202 the NRF stores the NF configuration file.
  • Step 203 the NRF sends an NF registration response to NF1.
  • the NF registration response is used to notify the NF that the registration is successful.
  • the NF discovery process includes steps 204 to 205 .
  • Step 204 NF2 sends an NF discovery request message to the NRF, carrying the condition information of the NF to be searched, such as NF type (NF type).
  • NF type NF type
  • Step 205 the NRF sends an NF discovery response to NF2, carrying the information of the NF instance that meets the conditions, such as the NF ID (NF ID) or the NF IP address.
  • NF ID NF ID
  • NF IP address the information of the NF instance that meets the conditions
  • the NF update process includes steps 206a to 210.
  • Step 206a NF2 sends an NF state subscription request to the NRF, carrying the NF instance, for requesting to subscribe to the state information of the NF instance.
  • NF2 subscribes to the NRF for the state information of a certain NF instance (the following is an example of subscribing to the state information of NF1), and the subsequent NRF finds that the state information of the NF instance has changed, the NRF will send the updated information of the NF instance to NF2. status information.
  • Step 206b NRF sends NF status subscription response to NF2.
  • the NF state subscription response is used to notify the NF state subscription success.
  • Step 207 NF1 sends an NF update request to the NRF, carrying the updated NF configuration file.
  • Step 208 the NRF updates the NF configuration file.
  • the NRF updates the stored NF configuration file according to the received updated NF configuration file.
  • Step 209 the NRF sends an NF update response to NF1.
  • the NF update response is used to indicate that the NF configuration file was updated successfully.
  • Step 210 the NRF sends the NF state change notification to the NF2, carrying the updated NF configuration file.
  • the NRF sends the NF state change notification to NF2 that has previously subscribed to the state information of NF1.
  • FIG. 3 it is a schematic diagram of the workflow of training NWDAF and inferring NWDAF under the separation architecture of training and inference.
  • the functions of each network element are described as follows:
  • NRF responsible for NF management, and provides interface services including NF registration/deregistration/update, NF status subscription/notification, etc.
  • NWDAF responsible for model training, and the trained model can be used by other NWDAFs (such as inference NWDAFs).
  • Inference NWDAF responsible for model inference, use inference results for data analysis, and output data analysis results.
  • NF Responsible for a specific business function, and can call the service of inferring NWDAF to obtain data analysis results.
  • Step 301 train the NWDAF to send the NF registration request to the NRF, carrying the NF configuration file.
  • the NF configuration file includes information such as NF type, NF service name (NF Service), and analysis type identifier (Analytics ID).
  • the NF type may be NWDAF.
  • the NF service name may be ModelProvision.
  • the analysis type identifier is used to indicate a specific analysis type provided by the training NWDAF, such as Service Experience, Network Performance, UE Mobility, etc.
  • Step 302 the NRF stores the NF configuration file.
  • Step 303 the NRF sends the NF registration response to the training NWDAF.
  • the NF registration response is used to notify the training NWDAF that the registration is successful.
  • Step 304 infer that the NWDAF sends an NF discovery request to the NRF, carrying the NF configuration file.
  • the carried NF configuration file contains NF type (such as NWDAF), NF service name (such as ModelProvision), and carries Analytics ID
  • NF type such as NWDAF
  • NF service name such as ModelProvision
  • Step 305 the NRF sends the NF discovery response to the inferred NWDAF, which carries the NWDAF instance.
  • the carried NWDAF instance is an instance of the training NWDAF, which can be represented by the ID or IP address of the training NWDAF.
  • steps 301 to 305 are optional steps. For example, if it is inferred that the NF configuration information for training the NWDAF is configured on the NWDAF, steps 301 to 305 may not be performed.
  • Step 306 the inference NWDAF sends a model request to the training NWDAF, which carries the Analytics ID.
  • the inferred NWDAF can send a model request to the training NWDAF based on the ID or IP address of the training NWDAF obtained from the NRF, and the carried Analytics ID is used to indicate the request to obtain the model corresponding to the Analytics ID.
  • Step 307 the training NWDAF sends a model response to the inference NWDAF, carrying model information.
  • the model (also known as machine learning model, Machine Learning Model, ML Model) information is used to describe the method of determining the sample output data according to the sample input data
  • the model information may include but is not limited to one or more of the following information : The feature type corresponding to the input data, the feature extraction method (functional relationship) of the feature type corresponding to the input data, the type corresponding to the output data (category label, continuous value, etc.), the type of algorithm used by the model, and the type of the model (classification, regression, etc.) , clustering, etc.), the parameters of the model.
  • the model can determine whether the sample is a cat or a dog according to the input data of the shape sample of an unknown animal.
  • the feature types of the input data can be animal weight, hair length, and bark, and the input data corresponds to
  • the extraction method of animal weight can be maximum and minimum normalization
  • the corresponding type of output data is cat or dog
  • the type of algorithm used by the model can be deep neural network (DNN)
  • the category of the model is classification
  • the parameters of the model include, but are not limited to: the number of layers of the neural network, the activation function used by each layer, and one or more function parameter values corresponding to the activation function of each layer.
  • model information such as first model information, second model information, etc.
  • model information such as the first information of the model, the second information of the model, etc.
  • model information can refer to the information about the model description, and will not be repeated elsewhere.
  • the above steps 301 to 307 are the flow of training the NWDAF to provide model services.
  • the training NWDAF registers the NF configuration file with the NRF, and subsequently infers that the NWDAF can obtain the training NWDAF instance from the NRF, and then infers that the NWDAF can request the training NWDAF to obtain specific types of model information. That is, training NWDAF can provide model services to infer NWDAF.
  • Step 308 infer that the NWDAF sends an NF registration request to the NRF, carrying the NF configuration file.
  • the NF configuration file includes information such as NF type, NF service name (NF Service), and analysis type identifier (Analytics ID).
  • the NF type may be NWDAF.
  • the name of the NF service may be to provide analysis services (Analytics).
  • the analysis type identifier is used to indicate a specific analysis type provided by the training NWDAF, such as Service Experience, Network Performance, UE Mobility, etc.
  • Step 309 the NRF stores the NF configuration file.
  • Step 310 the NRF sends a NF registration response to the inferred NWDAF.
  • the NF registration response is used to inform the inferred NWDAF registration success.
  • Step 311 the NF sends an NF discovery request to the NRF, carrying the NF configuration file.
  • the NF refers to a NF consumer (NF consumer), such as SMF, AMF or UPF, etc.
  • NF consumer such as SMF, AMF or UPF, etc.
  • the carried NF configuration file contains the NF type (such as NWDAF), the NF service name (such as ModelProvision), and carries the Analytics ID
  • the NF discovery request is used to request to obtain the inferred NWDAF corresponding to the Analytics ID from the NRF.
  • Step 312 the NRF sends the NF discovery response to the NF, which carries the NWDAF instance.
  • the carried NWDAF instance is an instance of the inferred NWDAF, which can be represented by the ID or IP address of the inferred NWDAF.
  • steps 308 to 312 are optional. For example, if the inferred NWDAF NF configuration information is configured on the NF, steps 308 to 312 may not be executed.
  • Step 313 the NF sends the analysis subscription to the inferred NWDAF, which carries the Analytics ID.
  • the NF may send an analysis subscription to the inferred NWDAF based on the ID or IP address of the inferred NWDAF obtained from the NRF, and the carried Analytics ID is used to instruct the subscription to obtain the data analysis result corresponding to the Analytics ID.
  • Step 314 infer that the NWDAF sends an analysis result notification to the NF, carrying the data analysis result.
  • the above steps 308 to 314 are the processes of inferring that the NWDAF provides analysis services.
  • the inferred NWDAF registers the NF configuration file with the NRF, and the subsequent NF can obtain the inferred NWDAF instance from the NRF, and then the NF can request the inferred NWDAF to obtain specific types of data analysis results. That is, it is inferred that NWDAF can provide data analysis services to NF.
  • steps 313 to 314 can be replaced by the following steps 313' to 314':
  • Step 313' the NF sends an analysis request to the inferred NWDAF, which carries the Analytics ID.
  • the NF may send an analysis subscription to the inferred NWDAF based on the ID or IP address of the inferred NWDAF obtained from the NRF, and the carried Analytics ID is used to instruct the request to obtain the data analysis result corresponding to the Analytics ID.
  • Step 314' infer that the NWDAF sends an analysis result response to the NF, carrying the data analysis result.
  • steps 313' to 314' it is necessary to actively send an analysis request each time, and it is inferred that the NWDAF only sends the data analysis results to the NF, and the above steps 313 to 314 only need to subscribe once, and it is inferred that the NWDAF is generating new data analysis results. Actively send data analysis results to NF.
  • the inference NWDAF can determine the inference result locally based on the inferred data, and then determine the use effect of the model based on the actual results of the inferred data and the inference results ( That is, the model performance evaluation result), infer that NWDAF determines the performance of the machine learning model according to the use effect, but training NWDAF in the training-inference separation scenario cannot perceive the use effect of the model in inference NWDAF, and infer NWDAF has no ability to perform model training, so , it is impossible to retrain and update the model when the performance of the model is degraded in the prior art, and it is impossible to ensure that the performance of the model is always good during the running process. If it is inferred that NWDAF continues to use the degraded model for data analysis, it may lead to inaccurate data analysis results and affect the model performance.
  • the embodiment of the present application proposes to establish a model performance monitoring and feedback mechanism to evaluate the performance of the model running in the inference NWDAF.
  • NWDAF can use the new model with good performance obtained by retraining to update (or replace) the model to ensure the use effect of the model.
  • the monitoring, feedback, retraining and updating mechanisms can be implemented by NRF, or by direct interaction between training NWDAF and inferring NWDAF.
  • the system architecture applied by the embodiments of the present application is the eNA architecture.
  • the embodiments of the present application are directed to scenarios in which model training and inference functions are deployed separately, that is, the training functions and inference functions are deployed in different NWDAF instances.
  • FIG. 4 it is a schematic diagram of a network architecture to which this embodiment of the present application is applied.
  • Training NWDAF, inferring NWDAF and NF all need to be registered in NRF through the Nnrf interface service.
  • the inference NWDAF requests the model from the training NWDAF through the Nnwdaf interface service, and the NF requests the data analysis results from the inference training NWDAF through the Nnwdaf interface service.
  • FIG. 5 a schematic flowchart of a method for ensuring model validity in a training-inference separation scenario provided by an embodiment of the present application.
  • the first embodiment considers updating registration information through NRF to implement performance monitoring and model updating. Mainly involved:
  • Model performance monitoring and feedback Infer the registration information of NWDAF at NRF to add model status information, infer NWDAF to monitor model performance, update the model status information through NRF when it is judged that the model needs to be retrained, and NRF notifies the training NWDAF model status update, Trigger training NWDAF to retrain the model.
  • Model update train the NWDAF registration information at the NRF to add the model index information, train the NWDAF and retrain the new model and update the model index information through the NRF, and the NRF informs the inferred NWDAF that a new model can be used. New model and complete model update.
  • Step 501 train the NWDAF to register with the NRF.
  • the NF configuration file includes NF type, NF service name (for example, NF Service), analysis type identifier (for example, Analytics ID) and other information, and also includes model index information .
  • the model index information may be a model version number (eg, version), location information (eg, location), or a Uniform Resource Locator (Uniform Resource Locator, URL), and the like. Among them, version represents the model version, location or URL represents the storage location of the model, and any one of the three can be used.
  • the model index information is location information or a URL
  • the location information or URL may also include a model version.
  • the location information may be an IP address.
  • the NRF stores the NF profile and sends the NF registration response to the training NWDAF.
  • Step 502 infer that the NWDAF is registered with the NRF.
  • NWDAF sends NF registration request to NRF, carrying NF configuration file
  • the NF configuration file includes NF type, NF service name (for example, NF Service), analysis type identifier (for example, Analytics ID) and other information, and also includes model state information .
  • the model status information is used to indicate the model usage status.
  • the selectable values of the model state information include but are not limited to:
  • Stop 'stopped' Indicates that the model is closed and has stopped providing services.
  • the model status information carried in the NF registration request is 'null', that is, it is inferred that there is currently no model available on NWDAF.
  • the NRF stores the NF profile and sends the NF registration response to the inferred NWDAF.
  • Step 503 the inference NWDAF finds the training NWDAF, and requests the training NWDAF to obtain model information.
  • the inferred NWDAF can obtain model information from the trained NWDAF.
  • Step 504 infer that the NWDAF subscribes to the NRF for the state of the training NWDAF.
  • the NRF When the NF profile registered in the NRF for the subsequent training NWDAF is updated, the NRF notifies the inferred NWDAF.
  • Step 505 train the NWDAF to subscribe to the NRF to infer the state of the NWDAF.
  • the NRF notifies the training NWDAF when the NF profile registered in the NRF is subsequently inferred.
  • Step 506 infer that the NWDAF sends an NF update request to the NRF, carrying the updated NF configuration file.
  • the updated NF configuration file carries at least updated model state information, and the updated model state information may be "ok", for example.
  • the updated NF configuration file also carries the Analytics ID, which is used to identify the model to be updated.
  • the updated NF configuration file also carries the NF type, the NF service name (NF Service), and the like.
  • Step 507 the NRF updates the NF configuration file.
  • the NRF updates the stored NF configuration file according to the received updated NF configuration file.
  • Step 508 the NRF sends a NF update response to the inferred NWDAF.
  • the NF update response is used to inform the inferred NWDAF that the NF configuration file update was successful.
  • Step 509 the NRF sends the NF state update notification to the training NWDAF, which carries the updated model state information.
  • the updated model state information may be, for example, 'ok'.
  • the NF state update notification further carries indication information, which is used to indicate that the update type is model state information update.
  • the training NWDAF subscribes the state of the inferred NWDAF to the NRF in the above step 505, when the NF configuration file of the inferred NWDAF stored in the NRF is updated, the NRF notifies the training NWDAF.
  • Step 510 infer that the NWDAF determines that the model needs to be retrained.
  • the judgment basis can be that the evaluation result of the model performance does not meet the model performance requirements (for example, the model accuracy drops below 80%, of which 80% is the model accuracy requirement), or it can be the business key performance indicators (Key Performance Indicator, KPI) reported by NF ) does not meet the KPI requirements (for example, the KPI falls below the KPI requirements).
  • KPI Key Performance Indicator
  • this step 510 occurs during the running process of the model in the inferred NWDAF, and the occurrence time is not fixed.
  • Step 511 infer that the NWDAF sends an NF update request to the NRF, carrying the updated NF configuration file.
  • the updated NF configuration file carries at least updated model state information, and the updated model state information may be 'limited', for example.
  • the updated NF configuration file also carries the Analytics ID, which is used to identify the model to be updated.
  • the updated NF configuration file also carries the NF type, the NF service name (NF Service), and the like.
  • the NRF updates the stored NF profile and then sends the NF update response to the inferred NWDAF.
  • Step 512 the NRF sends the NF state update notification to the training NWDAF, which carries the updated model state information.
  • the updated model state information may be 'limited'.
  • the NF state update notification further carries indication information, which is used to indicate that the update type is model state information update.
  • the training NWDAF subscribes the state of the inferred NWDAF to the NRF in the above step 505, when the NF configuration file of the inferred NWDAF stored in the NRF is updated, the NRF notifies the training NWDAF.
  • Step 513 Train the NWDAF to start retraining the model.
  • Training NWDAF starts retraining of the model, and obtains the trained model and the corresponding model index information, such as model version number, location information or URL, etc.
  • Step 514 train the NWDAF to send an NF update request to the NRF, carrying the updated NF configuration file.
  • the updated NF configuration file carries at least updated model index information, and the updated model index information may be, for example, updated model version information, updated model location information, or updated model URL, and the like.
  • the updated NF configuration file also carries the Analytics ID, which is used to identify the model to be updated.
  • the updated NF configuration file also carries the NF type, the NF service name (NF Service), and the like.
  • the NRF updates the stored NF profile and then sends the NF update response to the training NWDAF.
  • Step 515 the NRF sends the NF state update notification to the inferred NWDAF, which carries the updated model index information.
  • the updated model index information may be, for example, updated model version information, updated model location information, or updated model URL, and the like.
  • the NF state update notification further carries indication information, which is used to indicate that the type of update is model index information update.
  • the NRF Since in the above step 504, the inferred NWDAF subscribes to the NRF for the state of the training NWDAF, the NRF notifies the inferred NWDAF after the NF configuration file of the training NWDAF stored in the NRF is updated.
  • Step 516 the inference NWDAF sends a model request to the training NWDAF, carrying the Analytics ID and updated model index information.
  • the Analytics ID is used to indicate the model corresponding to that Analytics ID.
  • Step 517 the training NWDAF sends a model response to the inference NWDAF, carrying model information.
  • the model information includes model information corresponding to the updated model index information, that is, model information corresponding to the acquired new model.
  • the model information carried in the model response may be parameter values of the new model, or a new model (such as a model file or an image file containing the model), or a new model address (such as a URL or IP address).
  • a new model such as a model file or an image file containing the model
  • a new model address such as a URL or IP address
  • the model file is a model persistent file saved by a third-party framework, such as a model file in .pb format saved by the artificial intelligence framework TensorFlow.
  • a model image file is an image package that contains a model, which can contain the model file and several other files related to the use of the model.
  • the NWDAF can further obtain new model information directly according to the address information, without performing steps 516-517.
  • NWDAF obtains a file containing new model information (such as a file containing the values of the parameter items of the new model, or a new model file, or an image containing the new model through the File Transfer Protocol (FTP) according to the URL. document).
  • FTP File Transfer Protocol
  • the model information carried in the model response may be the value of the parameter item of the new model, or the new model (the model file or the image containing the model), or the new model.
  • the address of the model (such as a URL or IP address). If the model information carried in the model response is the address of the new model (such as URL or IP address), it is inferred that NWDAF can further obtain new model information according to the address information.
  • Step 518 infer that NWDAF performs model update.
  • NWDAF updates or replaces the old model being used according to the received new model information.
  • NWDAF performs a local test on the new model information before the model is updated, and then updates or replaces it after the test is passed.
  • Step 519 infer that the NWDAF sends an NF update request to the NRF, carrying the updated NF configuration file.
  • the updated NF configuration file carries at least updated model state information, for example, the updated model state information may be 'ok'.
  • the updated NF configuration file also carries the Analytics ID, which is used to identify the model to be updated.
  • the updated NF configuration file also carries the NF type, the NF service name (NF Service), and the like.
  • the NRF updates the stored NF profile and then sends the NF update response to the inferred NWDAF.
  • Step 520 the NRF sends the NF state update notification to the training NWDAF, which carries the updated model state information.
  • the updated model state information may be 'ok'.
  • the NF state update notification also carries indication information, which is used to indicate that the type of update is model state information update.
  • steps 511 to 520 are optional steps. For example, if the inferred NWDAF determines in step 510 that the model does not need to be retrained or the inferred NWDAF can tolerate a drop in model performance below the model performance requirement, steps 511 to 520 may not be performed.
  • the training NWDAF subscribes the state of the inferred NWDAF to the NRF in the above step 505, when the NF configuration file of the inferred NWDAF stored in the NRF is updated, the NRF notifies the training NWDAF.
  • the training NWDAF can be notified through the NRF to retrain the model.
  • the inferred NWDAF can use the new model to update or replace the old model to ensure the use of the model. Effect.
  • FIG. 6 a schematic flowchart of another method for ensuring model validity in a training-inference separation scenario provided by an embodiment of the present application.
  • the inferred NWDAF2 Since the inferred NWDAF2 also subscribes to the training NWDAF state, when the NWDAF is trained and then trained to obtain a new model, the inferred NWDAF2 will also receive notifications from the NRF. On the one hand, if the new model is better than the model in inferred NWDAF2, inferred NWDAF2 can use the new model to further improve the data analysis effect. On the other hand, since the model in inferred NWDAF2 does not have to be updated at this time, inferred NWDAF2 needs to obtain new The model needs to be updated only after it has been evaluated locally. If the inferred NWDAF2 finally decides not to update after acquiring the new model, part of the transmission resources is wasted. In this embodiment, the registration information of the training NWDAF is considered to further increase model performance information, including accuracy, required calculation amount, etc., to help other inferred NWDAFs that do not need to be updated temporarily to determine whether a new model needs to be requested.
  • Step 601 train the NWDAF to register with the NRF.
  • the model index information may be a model version number (version), location information (location), or a URL.
  • version represents the model version
  • location or URL represents the storage location of the model, and any one of the three can be used.
  • the model index information is location or URL, the location or URL can also contain version.
  • the model performance information is used to indicate the performance of the model, for example, it may include model accuracy, hardware capability information required to achieve the accuracy, calculation amount required for model inference, model inference duration, model size, and the like.
  • the NF configuration file may also contain information such as the algorithm used by the model, the artificial intelligence framework, and the input characteristics of the model.
  • the NRF stores the NF profile and sends the NF registration response to the training NWDAF.
  • Steps 602 to 613 are similar to steps 502 to 513 in the first embodiment above.
  • the related operations related to inferring NWDAF1 and the related operations of inferring NWDAF2 may refer to the related operations related to inferring NWDAF in the above steps 502 to 513 respectively.
  • steps 610 to 611 it is inferred that NWDAF1 determines that the model needs to be retrained, and then sends an NF update request to the NRF, thereby triggering the training NWDAF to start retraining the model.
  • Step 614 train the NWDAF to send an NF update request to the NRF, carrying the updated NF configuration file.
  • the updated NF configuration file carries at least updated model index information and updated model performance information
  • the updated model index information may be, for example, updated model version information, updated model location information, or updated model URL, and the like.
  • the updated NF configuration file also carries the Analytics ID, which is used to identify the model to be updated.
  • the updated model performance information may include, for example, model accuracy, hardware capability information required to achieve the accuracy, calculation amount required for model inference, model inference duration, model size, and the like.
  • the updated NF configuration file also carries the NF type, the NF service name (NF Service), and the like.
  • the NRF updates the stored NF profile and then sends the NF update response to the training NWDAF.
  • Step 615 the NRF sends a NF state update notification to the inferred NWDAF1 and the inferred NWDAF2, respectively, which carries the updated model index information and the updated model performance information.
  • the updated model index information may be, for example, updated model version information, updated model location information, or updated model URL, and the like.
  • the updated model performance information may include, for example, model accuracy, hardware capability information required to achieve the accuracy, calculation amount required for model inference, model inference duration, model size, and the like.
  • the NF state update notification further carries indication information, which is used to indicate that the type of update is model index information update and model performance information update.
  • inferred NWDAF1 and inferred NWDAF2 subscribe to NRF for the state of training NWDAF, respectively, so when the NF configuration file of training NWDAF stored in NRF is updated, NRF notifies inferred NWDAF1 and inferred NWDAF2.
  • Step 616 infer that NWDAF2 judges whether the model needs to be updated.
  • the inferred NWDAF2 Since the inferred NWDAF2 is not the trigger for model training, after the inferred NWDAF2 receives the updated model index information, it needs to determine whether the model needs to be updated.
  • NWDAF2 can determine whether the model needs to be updated according to its own computing power, model performance requirements, and received updated model performance information. Alternatively, the inferred NWDAF2 can also determine whether the model needs to be updated according to the performance status of the model being used and the received updated model performance information.
  • Step 617 infer that NWDAF1 sends a model request to the training NWDAF, carrying the Analytics ID and updated model index information.
  • the Analytics ID is used to indicate the model corresponding to that Analytics ID.
  • the model needs to be updated when the inferred NWDAF1 receives the updated model index information.
  • Step 618 the training NWDAF sends a model response to the inference NWDAF1, which carries the model information.
  • the model information includes model information corresponding to the updated model identifier, that is, model information corresponding to the acquired new model.
  • Step 619 infer that NWDAF1 performs model update.
  • NWDAF updates or replaces the old model being used according to the received new model information.
  • NWDAF performs a local test on the new model information before the model is updated, and then updates or replaces it after the test is passed.
  • the inferred NWDAF2 determines that the model needs to be updated, then the inferred NWDAF2 also needs to perform operations similar to the above steps 617 to 619 to request the training NWDAF to obtain the updated model information, and then according to Received new model information to update the old model in use. If in the above step 616, it is concluded that the NWDAF2 determines that the model does not need to be updated, then the model update process does not need to be performed.
  • the inferred NWDAF subscribed to the same model can determine whether to request a new model according to the model performance information, so as to avoid unnecessary model transmission and local evaluation process, thereby improving the efficiency of the model update process and saving resources.
  • FIG. 7 it is a schematic flowchart of another method for ensuring model validity in a training-inference separation scenario provided by an embodiment of the present application.
  • the third embodiment is based on the above-mentioned first embodiment, and considers a scenario in which multiple sub-models are required to cooperate to complete the analysis for the same Analytics ID.
  • the performance of any sub-model will degrade, and the performance of the model corresponding to the Analytics ID will be degraded. If the model monitoring is performed only based on the Analytics ID, the performance of the sub-model cannot be accurately located, and all sub-models corresponding to the Analytics ID will be monitored. Retraining and updating, and in fact, some sub-models may perform well and do not need updating, which leads to unnecessary training and updating.
  • This embodiment considers further adding a model ID (model ID) to represent each sub-model.
  • Step 701 train the NWDAF to register with the NRF.
  • the model index information may be a model version number (version), location information (location), or a URL. Among them, version represents the model version, location or URL represents the storage location of the model, and any one of the three can be used. Optionally, when the model index information is location or URL, the location or URL can also contain version.
  • the model identifier is used to uniquely identify a model, for example, the model identifier may consist of NWDAF address, PLMN ID, and modelID unique within a certain NWDAF range.
  • the NF configuration file can carry multiple model index information, and each model identifier corresponds to one model index information.
  • the NRF stores the NF profile and sends the NF registration response to the training NWDAF.
  • the updated NF configuration file may also carry multiple model identifiers, and each model identifier is used to identify one of the multiple updated models.
  • Step 702 infer that the NWDAF is registered with the NRF.
  • NWDAF sends an NF registration request to NRF, carrying the NF configuration file.
  • the NF configuration file includes information such as NF type, NF service name (NF Service), analysis type identifier (Analytics ID), as well as model state information and model identifier. Wherein, each model identifier corresponds to a model state information.
  • the model status information is used to indicate the usage status of the model corresponding to the model ID.
  • the selectable values of the model state information include but are not limited to:
  • the model status information carried in the NF registration request is 'null', that is, it is inferred that there is currently no model available on NWDAF.
  • model state information and model identifiers carried by the NF configuration file are as follows:
  • model state information and model identifier carried in the NF configuration file are as follows:
  • the model state information and model identifier carried by the NF configuration file may be a list, and the list includes multiple items of information, and each item of information includes a model state information and a model identifier.
  • the NRF stores the NF profile and sends the NF registration response to the inferred NWDAF.
  • Step 703 the inference NWDAF finds the training NWDAF, and requests the training NWDAF to obtain model information.
  • the inferred NWDAF can obtain model information from the trained NWDAF.
  • Step 704 infer that the NWDAF subscribes to the NRF for the state of the training NWDAF.
  • the NRF When the NF profile registered in the NRF for the subsequent training NWDAF is updated, the NRF notifies the inferred NWDAF.
  • Step 705 train the NWDAF to subscribe to the NRF to infer the state of the NWDAF.
  • the NRF notifies the training NWDAF when the NF profile registered in the NRF is subsequently inferred.
  • Step 706 infer that the NWDAF sends an NF update request to the NRF, carrying the updated NF configuration file.
  • the updated NF configuration file carries at least updated model state information, and the updated model state information may be 'ok', for example.
  • each model identifier corresponds to an updated model state information.
  • the updated NF configuration file also carries the Analytics ID, which is used to identify the model to be updated.
  • the updated NF configuration file also carries the NF type, the NF service name (NF Service), and the like.
  • the updated NF configuration file may also carry a model identifier, which is used to identify the updated model.
  • Step 707 the NRF updates the stored NF configuration file.
  • Step 708 the NRF sends a NF update response to the inferred NWDAF.
  • the NF update response is used to notify the NF that the configuration file update is successful.
  • Step 709 the NRF sends the NF state update notification to the training NWDAF, which carries the updated model state information.
  • each model identifier corresponds to an updated model state information.
  • the updated model state information may be, for example, 'ok'.
  • the NF state update notification further carries indication information, which is used to indicate that the update type is model state information update.
  • the NF state update notification may also carry a model identifier, which is used to identify the updated model.
  • the training NWDAF subscribes to the NRF for the state of the inferred NWDAF in the above step 705, when the NF configuration file of the inferred NWDAF stored in the NRF is updated, the NRF notifies the training NWDAF.
  • Step 710 infer that the NWDAF determines that the model needs to be retrained.
  • the judgment basis can be the evaluation result of the model performance (such as the decrease of the model accuracy), or the business KPI reported by the NF (such as the decrease of the KPI).
  • this step 710 occurs during the running process of the model in the inferred NWDAF, and the occurrence time is not fixed.
  • the judgment result may be that one or several sub-models need to be retrained.
  • a model corresponding to an Analytics ID there are a total of 10 sub-models, which are represented by model ID 1 to model ID10.
  • the judgment result of this step 710 is, for example: the sub-models corresponding to model ID 1 to model ID 3 need to be retrained, and the sub-models corresponding to model ID 4 to model ID 10 do not need to be retrained.
  • Step 711 infer that the NWDAF sends an NF update request to the NRF, carrying the updated NF configuration file.
  • the updated NF configuration file carries at least updated model state information, wherein each model identifier corresponds to one updated model state information.
  • the updated model state information may be 'limited', for example.
  • the updated NF configuration file also carries the Analytics ID, which is used to identify the model to be updated.
  • the updated NF configuration file also carries the NF type, the NF service name (NF Service), and the like.
  • the updated NF configuration file may also carry a model identifier, which is used to identify the updated model.
  • the NRF updates the stored NF profile and then sends the NF update response to the inferred NWDAF.
  • model identification carried in the updated NF configuration file in this step 711 is the identification information of the sub-model that needs to be retrained determined in the above step 710, and the updated model state information is the sub-model that needs to be retrained.
  • the updated model state information corresponding to the identification information of the model.
  • Step 712 the NRF sends the NF state update notification to the training NWDAF, which carries the updated model state information.
  • each model identifier corresponds to an updated model state information.
  • the updated model state information may be 'limited'.
  • the NF state update notification further carries indication information, which is used to indicate that the update type is model state information update.
  • the NF state update notification may also carry a model identifier, which is used to identify the updated model.
  • the training NWDAF subscribes to the NRF for the state of the inferred NWDAF in the above step 705, when the NF configuration file of the inferred NWDAF stored in the NRF is updated, the NRF notifies the training NWDAF.
  • Step 713 Train the NWDAF to start retraining the model.
  • Training NWDAF starts retraining of the model, and obtains the trained model and the corresponding model index information, such as model version number, location information or URL, etc.
  • the received sub-models that need to be trained are started and retrained. For example, if the received model identifiers are model ID 1 to model ID 3, the sub-models corresponding to model ID 1 to model ID 3 are retrained.
  • Step 714 train the NWDAF to send an NF update request to the NRF, carrying the updated NF configuration file.
  • the updated NF configuration file carries at least updated model index information, and each model identifier corresponds to an updated model state information.
  • the updated model index information may be, for example, updated model version information, updated model location information, or updated model URL, and the like.
  • the updated NF configuration file also carries the Analytics ID, which is used to identify the model to be updated.
  • the updated NF configuration file also carries the NF type, the NF service name (NF Service), and the like.
  • the updated NF configuration file may also carry a model identifier, which is used to identify the updated model.
  • the NRF updates the stored NF profile and then sends the NF update response to the training NWDAF.
  • Step 715 the NRF sends the NF state update notification to the inferred NWDAF, which carries the updated model index information.
  • the updated model index information may be, for example, updated model version information, updated model location information, or updated model URL, and the like.
  • the NF state update notification further carries indication information, which is used to indicate that the type of update is model index information update.
  • the NF state update notification also carries a model identifier, which is used to identify the updated model.
  • the inferred NWDAF subscribes to the NRF for the state of the training NWDAF, when the NF configuration file of the training NWDAF stored in the NRF is updated, the NRF notifies the inferred NWDAF.
  • Step 716 the inference NWDAF sends a model request to the training NWDAF, carrying the Analytics ID and updated model index information.
  • the Analytics ID is used to indicate the model corresponding to that Analytics ID.
  • the model ID is used to indicate the submodel in the model corresponding to the Analytics ID.
  • Step 717 the training NWDAF sends a model response to the inference NWDAF, carrying model information.
  • the model information includes model information corresponding to the updated model identifier, that is, model information corresponding to the acquired new model.
  • Step 718 infer that NWDAF performs model update.
  • NWDAF updates the old model (specifically, the corresponding sub-model that needs to be updated) in use according to the received new model information.
  • NWDAF performs a local test on the new model information before the model is updated, and then updates or replaces it after the test is passed.
  • Step 719 infer that the NWDAF sends an NF update request to the NRF, carrying the updated NF configuration file.
  • the updated NF configuration file carries at least updated model state information, and each model identifier corresponds to one updated model state information.
  • the updated model state information may be, for example, 'ok'.
  • the updated NF configuration file also carries the Analytics ID, which is used to identify the model to be updated.
  • the updated NF configuration file also carries the NF type, the NF service name (NF Service), and the like.
  • the NRF updates the stored NF profile and then sends the NF update response to the inferred NWDAF.
  • the updated NF configuration file may also carry a model identifier, which is used to identify the updated model.
  • Step 720 the NRF sends the NF state update notification to the training NWDAF, which carries the updated model state information.
  • each model identifier corresponds to an updated model state information.
  • the updated model state information may be 'ok'.
  • the NF state update notification further carries indication information, which is used to indicate that the update type is model state information update.
  • the NF state update notification may also carry a model identifier, which is used to identify the updated model.
  • step 714-step 720 are optional steps. For example, if the training NWDAF determines in step 713 that the model does not need to be retrained or the training NWDAF can tolerate the model performance falling below the model performance requirements or the training NWDAF currently does not have the ability to retrain the model (such as limited hardware resources), step 714-step 720 may not be executed.
  • the training NWDAF subscribes to the NRF for the state of the inferred NWDAF in the above step 705, when the NF configuration file of the inferred NWDAF stored in the NRF is updated, the NRF notifies the training NWDAF.
  • model identification also called sub-model identification
  • performance monitoring is performed according to the granularity of sub-models.
  • one Analytics ID corresponds to multiple sub-models
  • accurate execution model retraining and updating can be realized to avoid waste. Training and transfer resources.
  • FIG. 8 a schematic flowchart of another method for ensuring model validity in a training-inference separation scenario provided by an embodiment of the present application.
  • the above embodiments 1 to 3 consider implementing the information exchange between the training NWDAF and the inference NWDAF through NRF, and the fourth embodiment considers adding an operation on the interface between the training NWDAF and the inference NWDAF to directly exchange information.
  • Step 801 train the NWDAF to register with the NRF.
  • NWDAF to send NF registration request to NRF, carrying NF configuration file, which includes NF type, NF service name (NF Service), analysis type identifier (Analytics ID) and other information.
  • NF configuration file which includes NF type, NF service name (NF Service), analysis type identifier (Analytics ID) and other information.
  • the NRF stores the NF profile and sends the NF registration response to the training NWDAF.
  • Step 802a infer that the NWDAF sends an NF discovery request to the NRF, carrying the NF configuration file.
  • the carried NF configuration file contains the NF type (such as NWDAF), the NF service name (such as ModelProvision), and carries the Analytics ID
  • the NF discovery request is used to request to obtain the training NWDAF corresponding to the Analytics ID from the NRF.
  • Step 802b the NRF sends a NF discovery response to the inferred NWDAF, which carries the NWDAF instance.
  • the carried NWDAF instance is an instance of the training NWDAF, which can be represented by the ID or IP address of the training NWDAF.
  • Step 803a the inference NWDAF sends a model request to the training NWDAF, which carries the Analytics ID.
  • the inferred NWDAF can send a model request to the training NWDAF based on the ID or IP address of the training NWDAF obtained from the NRF, and the carried Analytics ID is used to indicate the request to obtain the model corresponding to the Analytics ID.
  • Step 803b the training NWDAF sends a model response to the inference NWDAF, carrying model information.
  • steps 801-803b are optional steps. For example, if it is inferred that the NF configuration file for training the NWDAF is configured in the NWDAF, then steps 801-803b may not be performed.
  • Step 804a the training NWDAF sends a model performance information subscription request to the inference NWDAF, which carries the Analytics ID, model performance indicators (such as Precision, Accuracy, Error Rate, Recall, Recall, F1 Score (F-Score), Mean Square Error (MSE), Root Mean Squared Error (RMSE), Root Mean Squared Logarithmic Error (RMSLE), Mean Absolute Error (Mean Absolute Error, MAE), model inference time, model robustness, model scalability, model interpretability) and reporting period.
  • model performance indicators such as Precision, Accuracy, Error Rate, Recall, Recall, F1 Score (F-Score), Mean Square Error (MSE), Root Mean Squared Error (RMSE), Root Mean Squared Logarithmic Error (RMSLE), Mean Absolute Error (Mean Absolute Error, MAE), model inference time, model robustness, model scalability, model interpretability) and reporting period.
  • the precision rate, accuracy rate, error rate, recall rate, and F1 score are used to indicate the performance of the model of classification type or annotation type.
  • Mean Squared Error, Root Mean Squared Error, Root Mean Squared Logarithmic Error, Mean Absolute Error are used to indicate the performance of the regression class model.
  • Model inference duration is used to indicate how long it takes for the model to make predictions.
  • Model robustness is used to indicate the ability of the model to handle missing and outliers.
  • Model scalability is used to indicate the ability to handle large datasets.
  • Model interpretability is used to indicate the comprehensibility of model prediction criteria. For example, decision tree models have high model interpretability due to the generated rules or tree structure, and neural network models have low model interpretability due to a large number of model parameters.
  • Step 804b the inference NWDAF sends a model performance information notification to the training NWDAF, which carries the Analytics ID, the model performance index and the value corresponding to the model performance index.
  • the inference NWDAF periodically sends a model performance information notification to the training NWDAF based on the reporting period.
  • the NWDAF can periodically report the model performance information to the training NWDAF.
  • the model performance information notification may further carry the model performance requirements of the inferred NWDAF for the model, and/or the data used by the inferred NWDAF for model evaluation, and the like.
  • the model performance requirements can assist the training NWDAF to judge whether retraining is required and whether the performance of the model obtained by retraining meets the inference NWDAF requirements. ), the actual network measurement value (network data) corresponding to the inference result, which can be used when training NWDAF to retrain the model.
  • steps 804a to 804b can also be replaced by the following steps 804a' to 804b'.
  • Step 804a' the training NWDAF sends a model performance information subscription request to the inference NWDAF, which carries the Analytics ID, model performance indicators (such as precision rate, accuracy rate, error rate, recall rate, F1 score, mean square error, root mean square error, RMS logarithmic error, mean absolute error, model inference duration, model robustness, model scalability, model interpretability), and performance thresholds.
  • model performance indicators such as precision rate, accuracy rate, error rate, recall rate, F1 score, mean square error, root mean square error, RMS logarithmic error, mean absolute error, model inference duration, model robustness, model scalability, model interpretability
  • Step 804b' infer that the NWDAF sends a model performance retraining notification to the training NWDAF, which carries the Analytics ID.
  • the NWDAF determines that the value corresponding to the model performance index reaches the performance threshold, and reports the model performance information notification to the training NWDAF, and the model performance retraining notification is used to trigger the training NWDAF to retrain the model. Training.
  • the above step 804b' may not carry the performance threshold value, and the performance threshold value may be determined by the inferred NWDAF by itself.
  • the model performance retraining notification in step 804b' may also be referred to as a model performance reaching a threshold value notification or a model performance information notification.
  • the model performance information notification may further carry the model performance requirements of the inferred NWDAF for the model, and/or the data used by the inferred NWDAF for model evaluation, and the like.
  • the model performance requirement can be a threshold value determined by the inferred NWDAF itself, which is used to assist the training NWDAF to determine whether retraining is required and whether the performance of the model obtained from the retraining meets the requirements of the inferred NWDAF.
  • the data used by the inferred NWDAF for model evaluation include: The input data of the model, the output data of the model (inference result), and the actual measured value of the network (network data) corresponding to the inference result can be used when training NWDAF to retrain the model.
  • steps 804a to 804b can also be replaced by the following steps 804a" to 804b".
  • Step 804a the training NWDAF sends a model performance information request to the inference NWDAF, which carries the Analytics ID, model performance indicators (such as precision rate, accuracy rate, error rate, recall rate, F1 score, mean square error, root mean square error, average root logarithmic error, mean absolute error, model inference duration, model robustness, model scalability, model interpretability).
  • model performance indicators such as precision rate, accuracy rate, error rate, recall rate, F1 score, mean square error, root mean square error, average root logarithmic error, mean absolute error, model inference duration, model robustness, model scalability, model interpretability).
  • Step 804b infer that the NWDAF sends a model performance information response to the training NWDAF, which carries the Analytics ID, the model performance index, and the value corresponding to the model performance index.
  • the training NWDAF may periodically send a model performance information request to the inference NWDAF, and each time the inference NWDAF receives a model performance information request, the model performance evaluation is performed based on the model performance indicators, and the training NWDAF is sent to the training NWDAF. Send the Model Performance Information response.
  • the model performance information response may also carry the model performance requirements of the inferred NWDAF for the model, and/or the data used by the inferred NWDAF for model evaluation, and the like.
  • the model performance requirements can assist the training of NWDAF to determine whether retraining is required and whether the performance of the model obtained by retraining meets the requirements of inference NWDAF. ), the actual network measurement value (network data) corresponding to the inference result, which can be used when training NWDAF to retrain the model.
  • steps 804a to 804b can also be replaced by the following steps 804a''' to 804b'''.
  • Step 804a''' the training NWDAF sends a model performance data subscription request to the inference NWDAF, which carries the Analytics ID and the reporting period.
  • Step 804b''' the inference NWDAF sends a model performance data notification to the training NWDAF, which carries the Analytics ID and model performance evaluation reference information.
  • the model performance evaluation reference information includes at least one of the input data of the model, the output data (inference result) of the model, or the actual measurement value of the network corresponding to the inference result.
  • the NWDAF periodically sends a model performance data notification to the training NWDAF based on the reporting period, that is, it is inferred that the NWDAF can periodically report the model performance evaluation reference information to the training NWDAF.
  • the actual network measurement value (network data) corresponding to the inference result can be collected from the existing network by the inference NWDAF and reported to the training NWDAF, or can be collected from the existing network by the training NWDAF.
  • model performance data notification may further carry the model performance requirements of the inferred NWDAF for the model.
  • steps 804a to 804b may also be replaced by the following steps 804a"" to 804b"".
  • Step 804a" the training NWDAF sends a model performance data request to the inference NWDAF, which carries the Analytics ID.
  • the model performance data request further includes a time range, which is used to indicate that performance data within the time range is requested.
  • Step 804b" the inference NWDAF sends a model performance data response to the training NWDAF, which carries the Analytics ID and model performance evaluation reference information.
  • the model performance evaluation reference information includes at least one of the input data of the model, the output data (inference result) of the model, or the actual network measurement (network data) value corresponding to the inference result.
  • the training NWDAF can send a model performance data request to the inference NWDAF, and the inference NWDAF sends a model performance data response to the training NWDAF, that is, the inference NWDAF sends the model performance evaluation reference information to the training NWDAF.
  • the performance evaluation reference information may be within a certain time range.
  • the actual network measurement value corresponding to the inference result can be collected from the existing network by the inference NWDAF and reported to the training NWDAF, or can be collected from the existing network by the training NWDAF.
  • model performance data response may also carry the model performance requirements of the inferred NWDAF for the model.
  • Training NWDAF can construct a test set and perform model performance evaluation according to the model performance evaluation reference information sent by inference NWDAF.
  • Step 805 train the NWDAF to determine to start model retraining.
  • the training NWDAF determines that the value corresponding to the model performance index reaches the performance threshold value preset by the training NWDAF or does not meet the model performance requirements for inferring the NWDAF, and then it is determined to start the model retraining.
  • the training NWDAF receives the model performance information notification and determines to start model retraining.
  • the training NWDAF determines that the value corresponding to the model performance index reaches the performance threshold value preset by the training NWDAF or does not meet the model performance requirements for inferring the NWDAF, then it is determined to start the model retraining .
  • the training NWDAF determines that the model performance reaches the performance threshold preset by the training NWDAF according to the model performance evaluation reference information. limit, or the model performance requirements for inferring NWDAF are not met, then it is determined to initiate model retraining.
  • Step 806 the training NWDAF sends a model update request to the inference NWDAF, carrying the Analytics ID and new model information.
  • the new model information in the model update request may be the value of a parameter item of the new model, or a new model file or an image file containing the new model, or an address (such as a URL or IP address) of the new model.
  • NWDAF can obtain a file containing the information of the new model according to the address, and the file can be a file containing the values of the parameter items of the new model, or a model file, or an image file containing the new model.
  • Step 807 the inference NWDAF sends a model update response to the training NWDAF.
  • Step 808 infer that NWDAF performs model update.
  • NWDAF updates or replaces the old model being used according to the received new model information.
  • NWDAF performs a local test on the new model information before the model is updated, and then updates or replaces it after the test is passed.
  • step 806-step 808 are optional steps. For example, if the training NWDAF determines in step 805 that the model does not need to be retrained or the training NWDAF can tolerate the performance of the model falling below the model performance requirements or the training NWDAF currently does not have the ability to retrain the model (such as limited hardware resources), step 806-step 808 may not be executed.
  • the training NWDAF sends a model performance subscription or model performance request to the inference NWDAF, so as to monitor the performance of the model in the inference NWDAF, and when the performance drops to meet the retraining conditions, the training NWDAF performs retraining in a timely manner, and sends the new model to the inference NWDAF.
  • the inference NWDAF is used for updating, which guarantees the model performance of the model in the inference NWDAF.
  • FIG. 9 a schematic flowchart of another method for ensuring model validity in a training-inference separation scenario provided by an embodiment of the present application.
  • this embodiment considers adding parameters indicating model performance information in the model update request, including accuracy, required calculation amount, etc., to help other inference NWDAFs that do not need updating temporarily to determine whether a new model needs to be requested.
  • Steps 901 to 905 are similar to steps 801 to 805 in the fourth embodiment above.
  • the related operations related to inferring NWDAF1 and the related operations of inferring NWDAF2 may refer to the related operations of inferring NWDAF in the above steps 802a to 804b, respectively.
  • the training of the NWDAF is to trigger the start of model retraining according to the model performance information notification or the model performance information response sent by the inference NWDAF1.
  • training the NWDAF requires informing the inference NWDAF to perform model updates.
  • the first solution is to infer NWDAF without distinction, that is, training NWDAF always sends the new model information obtained by training to all inference NWDAFs.
  • the scheme refers to the following steps 906a to 906b.
  • the second solution is to distinguish between different inferred NWDAFs and only send new model information to the inferred NWDAF that triggered the training NWDAF to perform model training.
  • the scheme refers to the following steps 907a to 907c.
  • Step 906a the training NWDAF sends a model update request to the inference NWDAF1, carrying the Analytics ID, new model information and model performance information.
  • Step 906b infer that NWDAF1 judges whether the model needs to be updated.
  • NWDAF1 can judge whether the model needs to be updated based on the model performance information and/or the local test result of the new model information. If it is determined that an update is required, the old model is updated or replaced with the new model information.
  • Step 906c the training NWDAF sends a model update request to the inference NWDAF2, carrying the Analytics ID, new model information and model performance information.
  • Step 906d infer that NWDAF2 judges whether the model needs to be updated.
  • NWDAF2 can judge whether the model needs to be updated based on the model performance information and/or the local test result of the new model. If it is determined that an update is required, the old model is updated or replaced with the new model information.
  • Step 907a the training NWDAF sends a model update request to the inference NWDAF1, carrying the Analytics ID and new model information.
  • Step 907b infer the NWDAF1 update model.
  • NWDAF1 updates or replaces the old model with the new model information after receiving the model update request.
  • NWDAF performs a local test on the new model information before the model is updated, and then updates or replaces it after the test is passed.
  • Step 907c the training NWDAF sends a model training completion notification to the inference NWDAF2, carrying the Analytics ID and model performance information.
  • Step 907d infer that NWDAF2 judges whether the model needs to be updated.
  • NWDAF2 can decide whether to update the model according to its own computing power, model performance requirements, and received model performance information.
  • Step 907e optionally, infer that NWDAF2 sends a model request to the training NWDAF, carrying the Analytics ID.
  • the Analytics ID is used to indicate the model corresponding to that Analytics ID.
  • Step 907f optionally (depending on whether step 907e is executed), the training NWDAF sends a model response to the inference NWDAF2, carrying the new model information.
  • Step 907g infer the NWDAF2 update model.
  • NWDAF2 updates the old model in use according to the received new model information.
  • NWDAF2 performs a local test on the new model information before the model is updated, and then updates or replaces it after the test is passed.
  • the inferred NWDAF using the same model can obtain the information of the new model, and determine whether to request a new model according to the model performance information, avoiding unnecessary model transmission and local evaluation process.
  • FIG. 10 a schematic flowchart of another method for ensuring model validity in a training-inference separation scenario provided by an embodiment of the present application.
  • This embodiment is based on the fourth embodiment, and considers a scenario in which multiple sub-models are required to cooperate with each other to complete the analysis for the same Analytics ID. Similar to the solution of the third embodiment, the present embodiment considers further increasing the model ID to identify each sub-model, trains NWDAF to assign different model IDs to the sub-models, and accurately monitors the performance of each sub-model through the model ID.
  • Step 1001 train the NWDAF to register with the NRF.
  • NWDAF to send NF registration request to NRF, carrying NF configuration file, which includes NF type, NF service name (NF Service), analysis type identifier (Analytics ID) and other information.
  • NF configuration file which includes NF type, NF service name (NF Service), analysis type identifier (Analytics ID) and other information.
  • the NRF stores the NF profile and sends the NF registration response to the training NWDAF.
  • Step 1002a infer that the NWDAF sends an NF discovery request to the NRF, carrying the NF configuration file.
  • the carried NF configuration file contains the NF type (such as NWDAF), the NF service name (such as ModelProvision), and carries the Analytics ID
  • the NF discovery request is used to request to obtain the training NWDAF corresponding to the Analytics ID from the NRF.
  • Step 1002b the NRF sends a NF discovery response to the inferred NWDAF, which carries the NWDAF instance.
  • the carried NWDAF instance is an instance of the training NWDAF, which can be represented by the ID or IP address of the training NWDAF.
  • Step 1003a the inference NWDAF sends a model request to the training NWDAF, which carries the Analytics ID.
  • the inferred NWDAF can send a model request to the training NWDAF based on the ID or IP address of the training NWDAF obtained from the NRF, and the carried Analytics ID is used to indicate the request to obtain the model corresponding to the Analytics ID.
  • Step 1003b the training NWDAF sends a model response to the inference NWDAF, carrying model information and a model identifier.
  • each model identifier corresponds to one model information.
  • the model information and the model identifier may be implemented in the form of a model list, that is, the model response carries a model list, and the model list includes the model information and the model identifier and the corresponding relationship between the model information and the model identifier.
  • the model list includes: ⁇ model information 1, model identification 1>, ⁇ model information 2, model identification 2>, . . .
  • steps 1001 to 1003b are optional steps. For example, if it is inferred that the NF profile for training the NWDAF is configured in the NWDAF, then steps 1001-1003b may not be performed.
  • Step 1004a the training NWDAF sends a model performance information subscription request to the inference NWDAF, which carries the Analytics ID, model performance indicators (such as precision rate, accuracy rate, error rate, recall rate, F1 score, mean square error, root mean square error, average root logarithmic error, mean absolute error, model inference duration, model robustness, model scalability, model interpretability), reporting period, and model identification.
  • model performance indicators such as precision rate, accuracy rate, error rate, recall rate, F1 score, mean square error, root mean square error, average root logarithmic error, mean absolute error, model inference duration, model robustness, model scalability, model interpretability
  • model performance information subscription request may carry multiple model identifiers, as well as model performance indicators and reporting periods corresponding to each model identifier.
  • the reporting period corresponding to each model identifier is the same, only one reporting period may be carried.
  • Step 1004b the inference NWDAF sends a model performance information notification to the training NWDAF, which carries the Analytics ID, the model performance index and the value corresponding to the model performance index.
  • the inference NWDAF is based on the reporting period, and periodically sends the model performance information notification corresponding to each sub-model to the training NWDAF.
  • the NWDAF can periodically report the model performance information corresponding to each sub-model to the training NWDAF.
  • the model performance information notification may further carry the model performance requirements of the inferred NWDAF for each sub-model, and/or the data used by the inferred NWDAF to evaluate each sub-model, and the like.
  • the model performance requirements can assist the training of NWDAF to determine whether retraining is required and whether the performance of the model obtained by retraining meets the requirements of inference NWDAF. ), the actual measurement value of the network corresponding to the inference result, which can be used when training NWDAF to retrain the model.
  • steps 1004a to 1004b can also be replaced by the following steps 1004a' to 1004b'.
  • Step 1004a' the training NWDAF sends a model performance information subscription request to the inference NWDAF, which carries the Analytics ID, model performance indicators (such as precision, accuracy, error, recall, F1 score, mean square error, root mean square error, RMS logarithmic error, mean absolute error, model inference duration, model robustness, model scalability, model interpretability), performance thresholds, and model identification.
  • model performance indicators such as precision, accuracy, error, recall, F1 score, mean square error, root mean square error, RMS logarithmic error, mean absolute error, model inference duration, model robustness, model scalability, model interpretability
  • model performance information subscription request may carry multiple model identifiers and model performance indicators and performance thresholds corresponding to each model identifier.
  • the performance threshold value corresponding to each model identifier is the same, only one performance threshold value may be carried.
  • Step 1004b' infer that the NWDAF sends a model performance retraining notification to the training NWDAF, which carries the Analytics ID.
  • NWDAF determines that the value corresponding to the model performance index of the sub-model reaches the performance threshold, and reports the model performance information notification corresponding to the sub-model to the training NWDAF, and the model performance retraining notification is used for To trigger the training NWDAF retrains the sub-model.
  • the above step 1004b' may not carry the performance threshold value, and the performance threshold value may be determined by the inferred NWDAF by itself.
  • the model performance retraining notification in this step 1004b' may also be referred to as a model performance reaching a threshold value notification or a model performance information notification.
  • the model performance information notification may further carry the model performance requirements of the inferred NWDAF for the sub-model, and/or the data used by the inferred NWDAF for model evaluation, and the like.
  • the model performance requirement can be a threshold value determined by the inferred NWDAF itself, which is used to assist the training NWDAF to determine whether retraining is required and whether the performance of the model obtained from the retraining meets the requirements of the inferred NWDAF.
  • the data used by the inferred NWDAF for model evaluation include: The input data of the model, the output data of the model (inference result), and the actual measurement value of the network corresponding to the inference result can be used when training NWDA to retrain the model.
  • steps 1004a to 1004b may also be replaced by the following steps 1004a" to 1004b".
  • Step 1004a the training NWDAF sends a model performance information request to the inference NWDAF, which carries the Analytics ID, model performance indicators (such as precision rate, accuracy rate, error rate, recall rate, F1 score, mean square error, root mean square error, average root logarithmic error, mean absolute error, model inference duration, model robustness, model scalability, model interpretability), and model identification.
  • model performance indicators such as precision rate, accuracy rate, error rate, recall rate, F1 score, mean square error, root mean square error, average root logarithmic error, mean absolute error, model inference duration, model robustness, model scalability, model interpretability
  • model performance information request may carry multiple model identifiers and model performance indicators corresponding to each model identifier.
  • Step 1004b infer that the NWDAF sends a model performance information response to the training NWDAF, which carries the Analytics ID, the model performance index, and the value corresponding to the model performance index.
  • the training NWDAF may periodically send a model performance information request to the inference NWDAF, and each time the inference NWDAF receives a model performance information request, the model performance evaluation is performed based on the model performance indicators, and the training NWDAF is sent to the training NWDAF. Send the model performance information response corresponding to the submodel.
  • the model performance information response may also carry the model performance requirements of the inferred NWDAF for each sub-model, and/or the data used by the inferred NWDAF for model evaluation, and the like.
  • the model performance requirements can assist the training of NWDAF to determine whether retraining is required and whether the performance of the model obtained by retraining meets the requirements of inference NWDAF. ), the actual network measurement value (network data) corresponding to the inference result, which can be used when training NWDAF to retrain the model.
  • steps 1004a to 1004b can also be replaced by the following steps 1004a''' to 1004b'''.
  • Step 1004a''' the training NWDAF sends a model performance data subscription request to the inference NWDAF, which carries the Analytics ID, the reporting period and the model identifier.
  • model performance data subscription request can carry multiple model identifiers and the reporting period corresponding to each model identifier.
  • Step 1004b"' the inference NWDAF sends a model performance data notification to the training NWDAF, which carries the Analytics ID and model performance evaluation reference information.
  • the model performance evaluation reference information includes at least one of input data of the model, output data of the model (inference result), or actual network measurement value (network data) corresponding to the inference result.
  • model performance evaluation reference information here may be multiple model performance evaluation reference information. Specifically, each model identifier corresponds to a model performance evaluation reference information.
  • NWDAF periodically sends model performance data notifications to the training NWDAF based on the reporting period, that is, it is inferred that the NWDAF can periodically report the model performance evaluation corresponding to each sub-model to the training NWDAF. Reference Information.
  • the actual network measurement value (network data) corresponding to the inference result can be collected from the existing network by the inference NWDAF and reported to the training NWDAF, or can be collected from the existing network by the training NWDAF.
  • model performance data notification may further carry the model performance requirements of the inferred NWDAF for each sub-model.
  • steps 1004a to 1004b may also be replaced by the following steps 1004a"" to 1004b"".
  • Step 1004a" the training NWDAF sends a model performance data request to the inference NWDAF, which carries the Analytics ID and the model identifier.
  • model performance data subscription request can carry multiple model identifiers.
  • the model performance data request further includes a time range, which is used to indicate that performance data within the time range is requested.
  • each model identifier may correspond to a time range.
  • Step 1004b" the inference NWDAF sends a model performance data response to the training NWDAF, which carries the Analytics ID and model performance evaluation reference information.
  • the model performance evaluation reference information includes at least one of the input data of the model, the output data of the model (inference result), and/or, or the actual measured value of the network (network data) corresponding to the inference result.
  • model performance evaluation reference information here may be multiple model performance evaluation reference information. Specifically, each model identifier corresponds to a model performance evaluation reference information.
  • the training NWDAF can send a model performance data request to the inference NWDAF, and the inference NWDAF sends a model performance data response to the training NWDAF, that is, the inference NWDAF sends the model performance evaluation reference information to the training NWDAF.
  • the performance evaluation reference information may be within a certain time range.
  • the actual network measurement value (network data) corresponding to the inference result can be collected from the existing network by the inference NWDAF and reported to the training NWDAF, or can be collected from the existing network by the training NWDAF.
  • model performance data response may also carry the model performance requirements of the inferred NWDAF for the model.
  • Training NWDAF can construct a test set and perform model performance evaluation according to the model performance evaluation reference information sent by inference NWDAF.
  • Step 1005 train the NWDAF to determine to start model retraining.
  • the training NWDAF determines that the value corresponding to the model performance index reaches the performance threshold value preset by the training NWDAF or does not meet the model performance requirements for inferring the NWDAF, then it is determined to start the model retraining.
  • the training NWDAF receives the model performance information notification and determines to start the model retraining.
  • the training NWDAF determines that the value corresponding to the model performance index reaches the performance threshold value preset by the training NWDAF or does not meet the model performance requirements for inferring the NWDAF, then it is determined to start the model retraining .
  • the training NWDAF is based on the model performance evaluation reference information to determine that the model performance reaches the performance threshold preset by the training NWDAF. limit, or the model performance requirements for inferring NWDAF are not met, then it is determined to initiate model retraining.
  • Step 1006 the training NWDAF sends a model update request to the inference NWDAF, carrying the Analytics ID, new model information and model identifier.
  • model update request may carry multiple model identifiers and new model information corresponding to each model identifier.
  • Step 1007 the inference NWDAF sends a model update response to the training NWDAF.
  • Step 1008 infer that NWDAF performs model update.
  • the NWDAF updates or replaces the old model in use (specifically, the old sub-model) according to the received new model information.
  • NWDAF performs a local test on the new model information before the model is updated, and then updates or replaces it after the test is passed.
  • step 1006-step 1008 are optional steps. For example, if the training NWDAF determines in step 805 that the model does not need to be retrained or the training NWDAF can tolerate the performance of the model falling below the model performance requirements or the training NWDAF currently does not have the ability to retrain the model (such as limited hardware resources), step 1006-step 1008 may not be executed.
  • FIG. 11 a schematic flowchart of another method for ensuring model validity in a training-inference separation scenario provided by an embodiment of the present application.
  • This embodiment considers periodic retraining by the trained NWDAF and informs the inferred NWDAF that new models are available. This embodiment is applicable to a scenario where the inferred NWDAF does not have an evaluation function, that is, the real-time feedback of the inferred NWDAF on the model performance cannot be obtained. To maintain the performance of the model, the trained NWDAF can be retrained periodically.
  • Step 1101 train the NWDAF to register with the NRF.
  • the NF configuration file includes NF type, NF service name (NF Service), analysis type identifier (Analytics ID) and other information, and also includes model index information.
  • the model index information may be a model version number (version), location information (location), or a uniform resource locator (Uniform Resource Locator, URL), and the like.
  • version represents the model version
  • location or URL represents the storage location of the model, any one of the three can be used.
  • the model index information is location or URL
  • the location or URL can also contain version.
  • the NRF stores the NF profile and sends the NF registration response to the training NWDAF.
  • Step 1102 the inferred NWDAF discovers the training NWDAF, and requests the training NWDAF to obtain model information.
  • the inferred NWDAF can obtain model information from the trained NWDAF.
  • Step 1103 infer that the NWDAF subscribes to the NRF for the state of training the NWDAF.
  • the NRF When the NF profile registered in the NRF for the subsequent training NWDAF is updated, the NRF notifies the inferred NWDAF.
  • Step 1104 train the NWDAF to periodically start model retraining.
  • training NWDAF can set a timer and retrain it every fixed time.
  • Step 1105 train the NWDAF to send an NF update request to the NRF, carrying the updated NF configuration file.
  • the updated NF configuration file carries at least updated model index information.
  • the updated NF configuration file also carries the NF type, the NF service name (NF Service), and the like.
  • the updated NF configuration file may also carry updated model performance information, such as model accuracy, hardware capability information required to achieve the accuracy, calculation amount required for model inference, model inference duration, model size etc.
  • Step 1106 the NRF updates the stored NF configuration file.
  • Step 1107 the NRF sends the NF update response to the training NWDAF.
  • the NF update response is used to notify the NF that the configuration file update is successful.
  • Step 1108 the NRF sends the NF state update notification to the inferred NWDAF, which carries the updated model index information.
  • the NF state update notification further carries indication information, which is used to indicate that the type of update is model index information update.
  • Step 1109 the inference NWDAF sends a model request to the training NWDAF, carrying the Analytics ID and updated model index information.
  • the Analytics ID is used to indicate the model corresponding to that Analytics ID.
  • Step 1110 the training NWDAF sends a model response to the inference NWDAF, carrying model information.
  • the model information includes model information corresponding to the updated model index information, that is, model information corresponding to the acquired new model.
  • Step 1111 infer NWDAF to update the model.
  • NWDAF updates the old model in use according to the new model information received.
  • NWDAF performs a local test on the new model information before the model is updated, and then updates or replaces it after the test is passed.
  • steps 1109 to 1111 are optional steps. For example, if the inference NWDAF after step 1108 can tolerate the degradation of model performance to the model performance requirement, steps 1109 to 1111 may not be performed.
  • steps 1104 to 1108 are performed periodically, so steps 1109 to 1111 are optional, because the training NWDAF is only responsible for periodic retraining, and whether to request a new model to update is up to the inference NWDAF itself Sure.
  • periodic retraining can be performed to ensure the performance of the model.
  • the NRF can send NF status update notifications to the multiple inferred NWDAFs, so that multiple inferred NWDAFs can be sent to the training NWDAFs Model requests, thereby enabling model updates for multiple inference NWDAFs.
  • the above step 1105 may also carry one or more model identifiers, and the above step 1108 , the one or more model identifiers can also be carried, and then the one or more model identifiers can be carried in the above step 1109, so as to realize the update of one or more sub-models in the inferred NWDAF.
  • FIG. 12 a schematic flowchart of another method for ensuring model validity in a training-inference separation scenario provided by an embodiment of the present application.
  • the eighth embodiment is the same as the seventh embodiment, that is, when the inferred NWDAF does not have an evaluation function, the training NWDAF performs periodic retraining, and sends a model update message to the inferred NWDAF.
  • Step 1201 train the NWDAF to register with the NRF.
  • NWDAF to send NF registration request to NRF, carrying NF configuration file, which includes NF type, NF service name (NF Service), analysis type identifier (Analytics ID) and other information.
  • NF configuration file which includes NF type, NF service name (NF Service), analysis type identifier (Analytics ID) and other information.
  • the NRF stores the NF profile and sends the NF registration response to the training NWDAF.
  • Step 1202 the inference NWDAF discovers the training NWDAF, and requests the training NWDAF to acquire the model.
  • the inferred NWDAF can obtain model information from the trained NWDAF.
  • Step 1203 the training NWDAF periodically starts model retraining.
  • training NWDAF can set a timer and retrain it every fixed time.
  • Step 1204 the training NWDAF sends a model update request to the inference NWDAF, carrying the Analytics ID and new model information.
  • the model update request may also carry performance information of the new model, such as model accuracy, hardware capability information required to achieve the accuracy, calculation amount required for model inference, model inference duration, model size, etc.
  • Step 1205 the inference NWDAF sends a model update response to the training NWDAF.
  • Step 1206, infer that NWDAF performs model update.
  • NWDAF updates the old model in use according to the new model information received.
  • NWDAF performs a local test on the new model information before the model is updated, and then updates or replaces it after the test is passed.
  • steps 1204 to 1206 are optional steps. For example, if the training NWDAF in step 1203 determines that the model performance evaluation result of the trained updated model is less than or equal to the performance evaluation result of the model provided by the training NWDAF to the inference NWDAF in step 1202, steps 1204-1206 may not be performed.
  • the training NWDAF can send a model update request to the multiple inferred NWDAFs, thereby realizing model update of the multiple inferred NWDAFs.
  • the above step 1204 can also carry one or more model identifiers, thereby realizing the inference of NWDAF. An update of one or more submodels in .
  • the relationship between the ninth embodiment and the above-mentioned first to eighth embodiments is that: the above-mentioned first to eighth embodiments are various specific implementations of the ninth embodiment.
  • FIG. 13 a schematic flowchart of a communication method provided by an embodiment of the present application is shown.
  • the first NWDAF in the ninth embodiment may be the training NWDAF in the first to eighth embodiments
  • the second NWDAF may be the inferred NWDAF1 in the first to eighth embodiments
  • the third NWDAF may be The inferred NWDAF2 in the first embodiment to the eighth embodiment above.
  • the method includes the following steps:
  • Step 1301 the first NWDAF sends third information to the second NWDAF. Accordingly, the second NWDAF receives the third information.
  • the third information includes a performance index of the model, and the performance index of the model is used to obtain an evaluation result of the performance of the model.
  • model performance indicators include one or more of the following: precision, accuracy, error, recall, F1 score, mean square error, root mean square error, root mean square logarithmic error, mean absolute error, Model inference time, model robustness, model scalability, model interpretability. That is, the second NWDAF evaluates the performance of the model in use according to the received performance index of the model, and then obtains a performance evaluation result, and generates a performance report of the model.
  • the third information further includes one or more of the following: an analysis type identifier, a model identifier, and a submodel identifier.
  • the analysis type identifier (Analytics ID) is used to indicate the analysis type of the model, such as Service Experience, Network Performance, UE Mobility, etc.
  • the ID of the model is used to identify the model.
  • the ID of the submodel is used to identify the submodels of this model. It should be noted that when the model has no sub-model, the third information may carry the identification of the model and does not need to carry the identification of the sub-model, or the third information neither carries the identification of the model nor the identification of the sub-model.
  • the model When the model has sub-models, it needs to carry the ID of the model and the ID of one or more sub-models at the same time. It should be noted that, when the third information carries the identifier of the sub-model, the performance index of the model is used to obtain the evaluation result of the performance of the sub-model of the model.
  • the third information further includes one or more of the following: reporting period and threshold information.
  • the reporting period is used to indicate the time to report the performance report of the model, that is, used to instruct the second NWDAF to report the performance report of the model to the first NWDAF based on the reporting period.
  • the threshold information is used to indicate the condition for reporting the performance report of the model, that is, when the evaluation result of the model obtained by the second NWDAF reaches the threshold value corresponding to the threshold information, the second NWDAF reports the performance report of the model to the first NWDAF .
  • this step 1301 is an optional step.
  • the above-mentioned third information may be pre-configured on the second NWDAF in advance, or the above-mentioned third information may be configured to the second NWDAF by other network elements.
  • Step 1302 the second NWDAF sends the first information to the first NWDAF. Accordingly, the first NWDAF receives the first information.
  • the first information includes a performance report of the model, where the performance report of the model is used to indicate the evaluation result of the performance of the model, or the performance report of the model is used to indicate that the evaluation result of the performance of the model does not meet the requirements of the performance index of the model.
  • the first information further includes one or more of the following information corresponding to the performance report of the model: time, region, slice.
  • the time refers to the time range for generating the performance report of the model
  • the area refers to the area range corresponding to the performance report of the model
  • the slice refers to the slice information corresponding to the performance report of the model.
  • Step 1303 the first NWDAF updates the first model information of the model according to the performance report of the model, and obtains the second model information of the model.
  • Step 1304 the first NWDAF sends the second information to the second NWDAF. Accordingly, the second NWDAF receives the second information.
  • the second information includes second model information.
  • the second information further includes one or more of the following: the identification of the model, the identification of the sub-model, the performance evaluation result of the model, the hardware capability information corresponding to the performance evaluation result of the model, the size of the model, and the inference duration of the model.
  • the hardware capability information corresponding to the performance evaluation result of the model refers to the hardware capability requirements required to run the model. The delay between receiving an input and producing an output.
  • each type of hardware capability information corresponds to one inference duration, and the stronger the hardware capability, the shorter the inference duration.
  • Step 1305 the second NWDAF updates the model according to the second information.
  • the second NWDAF replaces the first model information with the second model information to implement model update.
  • the first NWDAF may also send the second information to other NWDAFs (eg, the third NWDAF) other than the second NWDAF. That is, the first NWDAF is triggered by the second NWDAF to update the model to obtain the second model information, but the first NWDAF not only sends the second information to the second NWDAF, but also sends the second information to the third NWDAF, so as to realize the third
  • the NWDAF updates the model, thereby avoiding the need for the third NWDAF to request the model update from the first NWDAF separately, which can save signaling overhead.
  • the above step 1301 may specifically be: the first NWDAF sends the third information to the second NWDAF through the NRF
  • the above step 1302 may specifically be: the first NWDAF receives the first information from the second NWDAF through the NRF
  • the above Step 1304 may specifically be: the first NWDAF sends the second information to the second NWDAF through the NRF. That is, when there is no interface between the first NWDAF and the second NWDAF, the interaction between the first NWDAF and the second NWDAF can be implemented by using the NRF as an intermediate network element.
  • the second NWDAF when the second NWDAF cannot complete the model training, the second NWDAF can send a performance report of the model to the first NWDAF, so that the first NWDAF can update the model according to the performance report of the model, and obtain the second NWDAF of the model. model information, and send the second model information to the second NWDAF, so that the second NWDAF can update the model based on the second model information, so that the model can be trained in time when the model performance is degraded, thereby ensuring the model performance.
  • the model can be used based on network data corresponding to the service flow (such as the air interface quality of the terminal corresponding to the service flow at the base station side, the quality of service flow of the session of the terminal corresponding to the service flow at
  • the user plane manages the bandwidth, delay, jitter, etc. on the network element to evaluate the service experience of the service flow, and the network-side policy control network element (Policy Charging Function, PCF) can determine the service flow according to the service experience output result of the model. Whether the experience requirements are met, if not, you can adjust the QoS parameters of the service.
  • Policy Charging Function Policy Charging Function
  • the premise of the PCF adjusting QoS parameters is that the performance of the service experience model is good enough, otherwise, the service experience will be affected.
  • the MOS score MoS Opinion Score
  • the MOS score is required to be 3.0 points. If the actual MOS score of the service flow is 2.5 points, but the output MOS score of the model is divided into 3.5 points, then PCF will not adjust the QoS parameters of the service, which will lead to poor service experience. If the performance of the model is good enough, the output MOS score of the model should be 2.5 points, so that the PCF will adjust the service QoS parameters. Adjusted so that the MOS score reaches 3.0 or more. For this example, the performance of the model affects the business experience. And if the model performance continues to degrade, it may eventually deteriorate to the point where the model is completely unusable, resulting in extremely poor business experience or business interruption.
  • federated learning can realize cross-domain joint training of models without the original data in the domain, which can not only improve the efficiency of training, but most importantly, can avoid data aggregation to data through federated learning technology Security problems brought by the analysis center (for example, the original data is hijacked during transmission, the original data is misused by the data center, etc.).
  • Horizontal federated learning as a federated learning technology, is suitable for training data scenarios with "very high feature repetition, but large differences between data samples”.
  • the horizontal federation includes a central server (server) node and multiple edge client (client) nodes (for example, client node A, client node B, and client node K), in which the original data is distributed in each client.
  • client nodes for example, client node A, client node B, and client node K
  • server nodes do not have raw data
  • client nodes are not allowed to send raw data to server nodes.
  • x is the sample data
  • y is the label data corresponding to the sample data.
  • Each sample data in horizontal federated learning includes a label, that is, the label and the data are stored together.
  • the data analysis module on each client node can train its own model according to the linear regression algorithm, which is called a sub-model, namely:
  • h(x i ) ⁇ A x i A
  • h(x j ) ⁇ B x i B
  • h(x K ) ⁇ K Kx k K .
  • the objective function of sub-model training (the entire training process is to minimize the value of the above loss function) is:
  • N I represents the number of samples, represents the local gradient value.
  • the server node After the server node receives the above information, it aggregates the gradient as follows:
  • the server node sends the aggregated gradient to each client node participating in the training, and then the client node locally updates the model parameters, as follows:
  • the server node can control the end of the training through the number of iterations, such as training 10,000 times to terminate the training, or control the end of the training by setting the threshold of the loss function, such as when L I ⁇ 0.0001, the training ends.
  • each client node will keep the same model (which can be from the server node, or it can be obtained locally based on local personalization from the server node) for local inference.
  • horizontal federated learning and NWDAF can be combined to realize the model training and updating process.
  • the first NWDAF also called Server NWDAF
  • the second NWDAF also called Client NWDAF
  • the first NWDAF can train models or aggregate models
  • the second NWDAF also called Client NWDAF
  • FIG. 14( b ) it is a schematic flowchart of another communication method provided by an embodiment of the present application. The method includes the following steps:
  • Step 1401 the first NWDAF registers with the NRF.
  • the first NWDAF sends an NF registration request to the NRF, carrying the NF configuration file, where the NF configuration file includes information such as the NF type, the NF service name (NF Service, such as ModelProvision), and the analysis type identifier (Analytics ID).
  • NF Service such as ModelProvision
  • Analytics ID the analysis type identifier
  • the NRF stores the NF configuration file and sends the NF registration response to the first NWDAF.
  • Step 1402 the second NWDAF registers with the NRF.
  • the second NWDAF sends an NF registration request to the NRF, carrying the NF configuration file, where the NF configuration file includes information such as the NF type, the NF service name (NF Service, such as ModelUpdate), and the analysis type identifier (Analytics ID).
  • the NF configuration file includes information such as the NF type, the NF service name (NF Service, such as ModelUpdate), and the analysis type identifier (Analytics ID).
  • the NRF stores the NF configuration file and sends the NF registration response to the second NWDAF.
  • Step 1403 the second NWDAF sends an NF discovery request to the NRF, carrying the NF configuration file.
  • the carried NF configuration file contains NF type (such as NWDAF), NF service name (NF Service, such as ModelProvision, and carries Analytics ID, then the NF discovery request is used to request to obtain the Server NWDAF corresponding to the Analytics ID from NRF.
  • NF type such as NWDAF
  • NF service name such as ModelProvision
  • Step 1404 the NRF sends the NF discovery response to the second NWDAF, which carries the NWDAF instance.
  • the carried NWDAF instance is an instance of Server NWDAF, which can be represented by the ID or IP address of Server NWDAF.
  • Step 1405 the first NWDAF sends an NF discovery request to the NRF, carrying the NF configuration file.
  • the carried NF configuration file contains the NF type (such as NWDAF), the NF service name (NF Service, such as ModelUpdate), and carries the Analytics ID, then the NF discovery request is used to request to obtain the Client NWDAF corresponding to the Analytics ID from the NRF.
  • NF type such as NWDAF
  • NF Service such as ModelUpdate
  • Step 1406 the NRF sends an NF discovery response to the first NWDAF, which carries the NWDAF instance.
  • the carried NWDAF instance is an instance of Client NWDAF, which can be represented by the ID or IP address of Client NWDAF.
  • the NF discovery response can contain one or more Client NWDAF instances.
  • step 1403-step 1404" and “step 1405-step 1406" can be executed.
  • it can be the horizontal federation training that Client NWDAF actively triggers to Server NWDAF, or it can be Server NWDAF actively triggers horizontal federation training to Client NWDAF.
  • steps 1401 to 1406 are optional steps. For example, if the NF configuration file of the second NWDAF is configured in the first NWDAF and/or the NF configuration file of the first NWDAF is configured in the second NWDAF, steps 1401 to 1406 may not be performed.
  • Step 1407 the second NWDAF sends a model subscription request to the first NWDAF, carrying the Analytics ID.
  • the model subscription request is used to subscribe the model index information corresponding to the Analytics ID to the first NWDAF.
  • Step 1408 the first NWDAF sends model notification 1 to the second NWDAF, carrying model index information 1.
  • the model index information 1 is the index information of the model corresponding to the Analytics ID.
  • the model notification is the model notification corresponding to the model subscription request in step 1707 .
  • the second NWDAF may acquire the first information of the corresponding model according to the model index information 1.
  • Step 1409 the first NWDAF sends a model subscription request to the second NWDAF, carrying model index information 1.
  • the model subscription request is used to request the second NWDAF to update the first information of the model corresponding to the model index information 1, and to subscribe to the updated model information.
  • Step 1410 the model is updated.
  • the second NWDAF performs local training by using the model information corresponding to the model index information 1, obtains the second information of the model, and determines the model index information 2 corresponding to the second information of the model.
  • Step 1411 the second NWDAF sends a model notification to the first NWDAF, carrying model index information 2.
  • the model notification is the model notification corresponding to the model subscription request in step 1409 .
  • Step 1412 the model is updated.
  • the first NWDAF uses the second information of the model corresponding to the model index information 2 to perform local training, obtains the third information of the model, and determines the model index information 3 corresponding to the third information of the model.
  • the second NWDAF in steps 1407-1410 may be instances of multiple Client NWDAFs
  • the first NWDAF may receive model index information from multiple second NWDAF instances in step 1411
  • the first NWDAF may The model index information obtains corresponding multiple model information, and aggregates and trains the multiple model information to obtain updated model information.
  • Step 1413 the first NWDAF sends a model notification to the second NWDAF, carrying model index information 3.
  • the model notification is the model notification corresponding to the model subscription request 1 in step 1407 .
  • the above steps 1410 to 1413 may be repeated, and the model index information keeps changing until the first NWDAF determines to stop the iteration.
  • the first NWDAF may send a model unsubscribe message to the second NWDAF, that is, cancel the model subscription request corresponding to step 1409, so as to stop iteration.
  • the model index information may include identification information, where the identification information is used to indicate information of a model corresponding to the model index information.
  • the model index information further includes one or more of the following: an analysis type identifier corresponding to the model, an identifier of the model, and version information of the information of the model.
  • both the first NWDAF and the second NWDAF can update the model to obtain new model information, and send the model index information corresponding to the new model information to the other party, so that the model can be repeatedly iterated and the model performance can be achieved. Gradually improve the performance of the model, and finally obtain a model with the best model performance, which can ensure the performance of the model.
  • the communication apparatus 1500 includes a transceiver unit 1510 and a processing unit 1520 .
  • the communication device is used to implement the steps corresponding to the first data analysis network element in the foregoing embodiments:
  • the transceiver unit 1510 is configured to receive first information from the second data analysis network element, where the first information includes a performance report of the model, where the performance report of the model is used to indicate an evaluation result of the performance of the model, or, The performance report of the model is used to indicate that the evaluation result of the performance of the model does not meet the requirements of the performance index of the model; and, used to send second information to the second data analysis network element, the second The information includes second model information for the model.
  • the processing unit 1520 is configured to update the first model information of the model according to the performance report of the model to obtain the second model information.
  • the transceiver unit 1510 is further configured to send third information to the second data analysis network element, where the third information includes the performance index of the model, the performance of the model Metrics are used to obtain evaluation results of the performance of the model.
  • the transceiver unit 1510 is further configured to send the second information to a third data analysis network element.
  • the transceiver unit 1510 configured to receive the first information from the second data analysis network element, specifically includes: being configured to receive information from the second data analysis network element through a network storage network element of the first information.
  • the transceiver unit 1510, configured to send the second information to the second data analysis network element specifically includes: being configured to send the second information to the second data analysis network element through a network storage network element.
  • the model performance indicators include one or more of the following: precision rate, accuracy rate, error rate, recall rate, F1 score, mean square error, root mean square error, root mean square pair Number error, mean absolute error, model inference time, model robustness, model scalability, model interpretability.
  • the third information further includes one or more of the following: an analysis type identifier, an identifier of the model, and an identifier of a sub-model, where the analysis type identifier is used to indicate the Analysis type.
  • the third information further includes one or more of the following: a reporting period and threshold information, where the reporting period is used to indicate the time for reporting the performance report of the model, and the threshold information Used to indicate the conditions under which the performance report of the model is reported.
  • the first information further includes one or more of the following information corresponding to the performance report of the model: time, region, slice.
  • the second information further includes one or more of the following: the identification of the model, the identification of the sub-model, the performance evaluation result of the model, the corresponding performance evaluation result of the model hardware capability information, the size of the model, and the inference duration of the model.
  • the communication device is configured to implement the steps corresponding to the second data analysis network element in the above embodiments:
  • the transceiver unit 1510 is configured to send first information to the first data analysis network element, where the first information includes a performance report of the model, and the performance report of the model is used to indicate the evaluation result of the performance of the model, or, the The performance report of the model is used to indicate that the evaluation result of the performance of the model does not meet the requirements of the performance index of the model; and, for receiving the second information from the first data analysis network element, the second The information includes second model information of the model, and the second information of the model is obtained by updating the first model information of the model according to the performance report of the model.
  • the processing unit 1520 is configured to update the model according to the second model information.
  • the transceiver unit 1510 is further configured to receive third information from the first data analysis network element, where the third information includes the performance index of the model, the Performance metrics are used to obtain evaluation results of the performance of the model.
  • the transceiver unit 1510 configured to send the first information to the first data analysis network element, specifically includes: sending the information to the first data analysis network element through a network storage network element. the first information.
  • the transceiver unit 1510 is configured to receive the second information from the first data analysis network element, and specifically includes: being configured to receive the second information from the first data analysis network element through a network storage network element.
  • the model performance indicators include one or more of the following: precision rate, accuracy rate, error rate, recall rate, F1 score, mean square error, root mean square error, root mean square pair Number error, mean absolute error, model inference time, model robustness, model scalability, model interpretability.
  • the third information further includes one or more of the following: an analysis type identifier, an identifier of the model, and an identifier of a sub-model, where the analysis type identifier is used to indicate the Analysis type.
  • the third information further includes one or more of the following: a reporting period and threshold information, where the reporting period is used to indicate the time for reporting the performance report of the model, and the threshold information Used to indicate the conditions under which the performance report of the model is reported.
  • the first information further includes one or more of the following information corresponding to the performance report of the model: time, region, slice.
  • the second information further includes one or more of the following: the identification of the model, the identification of the sub-model, the performance evaluation result of the model, the corresponding performance evaluation result of the model hardware capability information, the size of the model, and the inference duration of the model.
  • the above-mentioned communication device may further include a storage unit, which is used to store data or instructions (also referred to as codes or programs), and each of the above-mentioned units may interact or be coupled with the storage unit to implement corresponding methods or functions.
  • the processing unit 1520 may read data or instructions in the storage unit, so that the communication apparatus implements the methods in the above embodiments.
  • each unit in the above communication apparatus can all be implemented in the form of software calling through the processing element; also can all be implemented in the form of hardware; some units can also be implemented in the form of software calling through the processing element, and some units can be implemented in the form of hardware.
  • each unit can be a separately established processing element, or can be integrated in a certain chip of the communication device to realize, in addition, it can also be stored in the memory in the form of a program, which can be called and executed by a certain processing element of the communication device. function of the unit.
  • each step of the above method or each of the above units may be implemented by an integrated logic circuit of hardware in the processor element or implemented in the form of software being invoked by the processing element.
  • a unit in any of the above communication devices may be one or more integrated circuits configured to implement the above method, such as: one or more application specific integrated circuits (ASICs), or, an or multiple microprocessors (digital singnal processors, DSP), or, one or more field programmable gate arrays (FPGA), or a combination of at least two of these integrated circuit forms.
  • ASICs application specific integrated circuits
  • DSP digital singnal processors
  • FPGA field programmable gate arrays
  • a unit in the communication device can be implemented in the form of a processing element scheduler
  • the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processors that can invoke programs.
  • CPU central processing unit
  • these units can be integrated together and implemented in the form of a system-on-a-chip (SOC).
  • the communication apparatus includes: a processor 1610 and an interface 1630 , and optionally, the communication apparatus further includes a memory 1620 .
  • the interface 1630 is used to enable communication with other devices.
  • the method performed by the first data analysis network element or the second data analysis network element may be invoked through the processor 1610 to the memory (which may be the memory 1620 in the first data analysis network element or the second data analysis network element, or may be is implemented by the program stored in the external memory). That is, the first data analysis network element or the second data analysis network element may include a processor 1610, and the processor 1610 executes the first data analysis network element or the second data analysis network element in the above method embodiments by calling a program in the memory The method performed by the network element.
  • the processor here may be an integrated circuit with signal processing capability, such as a CPU.
  • the first data analysis network element or the second data analysis network element may be implemented by one or more integrated circuits configured to implement the above methods. For example: one or more ASICs, or, one or more microprocessor DSPs, or, one or more FPGAs, etc., or a combination of at least two of these integrated circuit forms. Alternatively, the above implementations may be combined.
  • the functions/implementation process of the transceiver unit 1510 and the processing unit 1520 in FIG. 15 can be implemented by the processor 1610 in the communication apparatus 1600 shown in FIG. 16 calling computer executable instructions stored in the memory 1620 .
  • the function/implementation process of the processing unit 1520 in FIG. 15 can be implemented by the processor 1610 in the communication device 1600 shown in FIG. 16 calling the computer-executed instructions stored in the memory 1620, and the function of the transceiver unit 1510 in FIG. 15
  • the implementation process can be implemented through the interface 1630 in the communication device 1600 shown in FIG. 16 .
  • the function/implementation process of the transceiver unit 1510 can be implemented by the processor calling program instructions in the memory to drive the interface 1630 .
  • At least one item (single, species) of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.
  • “Plurality" means two or more, and other quantifiers are similar.
  • the above-mentioned embodiments it may be implemented in whole or in part by software, hardware, firmware or any combination thereof.
  • software it can be implemented in whole or in part in the form of a computer program product.
  • the computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated.
  • the computer may be a general purpose computer, special purpose computer, computer network, or other programmable device.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server, or data center is by wire (eg, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.).
  • the computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server, a data center, or the like that includes an integration of one or more available media.
  • the usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)), and the like.
  • a general-purpose processor may be a microprocessor, or alternatively, the general-purpose processor may be any conventional processor, controller, microcontroller, or state machine.
  • a processor may also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a digital signal processor core, or any other similar configuration. accomplish.
  • the steps of the method or algorithm described in the embodiments of this application may be directly embedded in hardware, a software unit executed by a processor, or a combination of the two.
  • Software units can be stored in random access memory (Random Access Memory, RAM), flash memory, read-only memory (Read-Only Memory, ROM), EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM or this.
  • RAM Random Access Memory
  • ROM read-only memory
  • EPROM memory read-only memory
  • EEPROM memory electrically erasable programmable read-only memory
  • registers hard disk, removable disk, CD-ROM or this.
  • a storage medium may be coupled to the processor such that the processor may read information from, and store information in, the storage medium.
  • the storage medium can also be integrated into the processor.
  • the processor and storage medium may be provided in the ASIC.
  • the above-described functions described herein may be implemented in hardware, software, firmware, or any combination of the three. If implemented in software, the functions may be stored on, or transmitted over, a computer-readable medium in the form of one or more instructions or code.
  • Computer-readable media includes computer storage media and communication media that facilitate the transfer of a computer program from one place to another. Storage media can be any available media that a general-purpose or special-purpose computer can access.
  • Such computer-readable media may include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other device that can be used to carry or store instructions or data structures and Other media in the form of program code that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor.
  • any connection is properly defined as a computer-readable medium, for example, if software is transmitted from a website site, server or other remote source over a coaxial cable, fiber optic computer, twisted pair, digital subscriber line (DSL) Or transmitted by wireless means such as infrared, wireless, and microwave are also included in the definition of computer-readable media.
  • DSL digital subscriber line
  • the discs and magnetic discs include compact discs, laser discs, optical discs, digital versatile discs (English: Digital Versatile Disc, DVD for short), floppy discs and Blu-ray discs. Disks usually reproduce data magnetically, while Discs usually use lasers to optically reproduce data. Combinations of the above can also be included in computer readable media.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Mobile Radio Communication Systems (AREA)
  • Radar Systems Or Details Thereof (AREA)

Abstract

本申请实施例提供通信方法、装置及系统。该方法包括：第一数据分析网元接收来自第二数据分析网元的第一信息，第一信息包括模型的性能报告；第一数据分析网元根据模型的性能报告更新模型的第一模型信息，获得模型的第二模型信息；第一数据分析网元向第二数据分析网元发送第二信息，第二信息包括第二模型信息。该方案，当第二数据分析网元无法完成模型训练时，向第一数据分析网元发送模型的性能报告，第一数据分析网元根据该模型的性能报告对模型进行更新得到模型的第二模型信息，并将第二模型信息发送给第二数据分析网元，使第二数据分析网元基于第二模型信息更新模型，从而实现在模型性能下降时及时对模型进行训练和更新，进而可以保证模型性能。

Description

通信方法、装置及系统 技术领域
本申请涉及通信技术领域，尤其涉及通信方法、装置及系统。
背景技术
机器学习模型的训练通常是通过学习某一组输入特征与输出目标之间的映射来进行的，通过优化某些损失函数来使得机器学习模型的输出结果(即预测值)与实际结果(即标签值/真实值)之间的误差最小化。当训练出最优模型后，利用该模型的输出对未来的情况进行预测。在理想情况下，假定未来将使用到的数据与模型训练期间所使用的数据类似，具体来说，可能假设训练时的输入特征和预测时的输入特征的分布保持恒定。但是实际中这种假设通常不成立，数据的特征会随着时间的推移由于网络部署变化、应用层业务要求变化、网络实际用户分布变化等而变化，因此，模型的性能(即泛化能力)会随着时间推移逐渐下降。具体的表现可能是，模型的准确率降低，即模型的预测值与真实值之间的误差变大。
以数据分析网元的训练功能和推断功能分离的场景为例，支持训练功能的数据分析网元(简称训练数据分析网元)无法感知支持推断功能的数据分析网元(简称推断数据分析网元)中的模型使用效果，且推断数据分析网元没有能力进行模型训练。因此，当模型性能下降时，若推断数据分析网元继续使用性能下降的模型进行数据分析，会导致数据分析结果不准确。
发明内容
本申请提供通信方法、装置及系统，用以实现在模型性能下降时及时对模型进行再次训练，从而保证模型性能。
第一方面，本申请实施例提供一种通信方法，包括：第一数据分析网元接收来自第二数据分析网元的第一信息，所述第一信息包括模型的性能报告，所述模型的性能报告用于指示所述模型的性能的评估结果，或者，所述模型的性能报告用于指示所述模型的性能的评估结果不满足所述模型的性能指标的要求；所述第一数据分析网元根据所述模型的性能报告更新所述模型的第一模型信息，获得所述模型的第二模型信息；所述第一数据分析网元向所述第二数据分析网元发送第二信息，所述第二信息包括所述第二模型信息。
基于上述方案，当第二数据分析网元无法完成模型训练时，则第二数据分析网元可以向第一数据分析网元发送模型的性能报告，从而第一数据分析网元可以根据该模型的性能报告对模型进行更新，得到模型的第二模型信息，并将第二模型信息发送给第二数据分析网元，使得第二数据分析网元可以基于第二模型信息更新模型，从而可以实现在模型性能下降时及时对模型进行训练，进而可以保证模型性能。
在一种可能的实现方法中，所述第一数据分析网元向所述第二数据分析网元发送第三信息，所述第三信息包括所述模型的性能指标，所述模型的性能指标用于获取所述模型的性能的评估结果。
基于上述方案，第一数据分析网元可以预先向第二数据分析网元发送模型的性能指标， 从而使得第二数据分析网元基于该模型的性能指标生成模型的性能报告，有助于帮助第一数据分析网元判断是否启动模型训练，并提升模型训练后的模型性能。
在一种可能的实现方法中，所述第一数据分析网元向第三数据分析网元发送所述第二信息。
基于上述方案，第一数据分析网元不仅可以向第二数据分析网元发送第二信息，还可以向其它使用该模型的网元，如第三数据分析网元发送该第二信息，使得第三数据分析网元也可以利用第二模型信息更新模型，提升模型使用效果。
在一种可能的实现方法中，所述第一数据分析网元接收来自第二数据分析网元的第一信息，包括：所述第一数据分析网元通过网络存储网元接收来自所述第二数据分析网元的所述第一信息。所述第一数据分析网元向所述第二数据分析网元发送第二信息，包括：所述第一数据分析网元通过网络存储网元向所述第二数据分析网元发送所述第二信息。
基于上述方案，可以通过网络存储网元作为中间网元，实现第一数据分析网元与第二数据分析网元之间的模型更新交互，这可以适用于第一数据分析网元与第二数据分析网元之间没有接口的场景。
第二方面，本申请实施例提供一种通信方法，包括：第二数据分析网元向第一数据分析网元发送第一信息，所述第一信息包括模型的性能报告，所述模型的性能报告用于指示所述模型的性能的评估结果，或者，所述模型的性能报告用于指示所述模型的性能的评估结果不满足所述模型的性能指标的要求；所述第二数据分析网元接收来自所述第一数据分析网元的第二信息，所述第二信息包括所述模型的第二模型信息，所述第二模型信息是根据所述模型的性能报告更新所述模型的第一模型信息得到的；所述第二数据分析网元根据所述第二模型信息，更新所述模型。
基于上述方案，当第二数据分析网元无法完成模型训练时，则第二数据分析网元可以向第一数据分析网元发送模型的性能报告，从而第一数据分析网元可以根据该模型的性能报告对模型进行更新，得到模型的第二模型信息，并将第二模型信息发送给第二数据分析网元，使得第二数据分析网元可以基于第二模型信息更新模型，从而可以实现在模型性能下降时及时对模型进行再次训练，进而可以保证模型性能。
在一种可能的实现方法中，所述第二数据分析网元接收来自所述第一数据分析网元的第三信息，所述第三信息包括所述模型的性能指标，所述模型的性能指标用于获取所述模型的性能的评估结果。
基于上述方案，第一数据分析网元可以预先向第二数据分析网元发送模型的性能指标，从而使得第二数据分析网元基于该模型的性能指标生成模型的性能报告，有助于帮助第一数据分析网元判断是否启动模型训练，并提升模型训练的准确性。
在一种可能的实现方法中，所述第二数据分析网元向第一数据分析网元发送第一信息，包括：所述第二数据分析网元通过网络存储网元向所述第一数据分析网元发送所述第一信息；所述第二数据分析网元接收来自所述第一数据分析网元的第二信息，包括：所述第二数据分析网元通过网络存储网元接收来自所述第一数据分析网元的第二信息。
基于上述方案，可以通过网络存储网元作为中间网元，实现第一数据分析网元与第二数据分析网元之间的模型更新交互，这可以适用于第一数据分析网元与第二数据分析网元之间没有接口的场景。
基于上述第一方面、或第一方面的任意实现方法、或第二方面、或第二方面的任意实 现方法：
在一种可能的实现方法中，所述模型性能指标包括以下一项或多项：精确率、准确率、错误率、召回率、F1分数、均方误差、均方根误差、均方根对数误差、平均绝对误差、模型推理时长、模型鲁棒性、模型可扩展性、模型可解释性。
在一种可能的实现方法中，所述第三信息还包括以下一项或多项：分析类型标识、所述模型的标识、子模型的标识，所述分析类型标识用于指示所述模型的分析类型。
在一种可能的实现方法中，所述第三信息还包括以下一项或多项：上报周期、门限信息，所述上报周期用于指示上报所述模型的性能报告的时间，所述门限信息用于指示上报所述模型的性能报告的条件。
基于上述方案，第一数据分析网元可以指示第二数据分析网元上报模型的性能报告的时间和/或条件，从而实现有条件的上报，可以节约资源开销。
在一种可能的实现方法中，所述第一信息还包括所述模型的性能报告对应的以下一项或多项信息：时间、区域、切片。
基于上述方案，当第一信息还包括模型的性能报告对应的时间、区域或切片，则可以提升第一数据分析网元执行模型再次训练后的模型性能。
在一种可能的实现方法中，所述第二信息还包括以下一项或多项：所述模型的标识、子模型的标识、所述模型的性能评估结果、所述模型的性能评估结果对应的硬件能力信息、所述模型的大小、所述模型的推理时长。
基于上述方案，第一数据分析网元将模型的性能评估结果、模型的性能评估结果对应的硬件能力信息、模型的大小或模型的推理时长中的一个或多个发送给第二数据分析网元，有助于第二数据分析网元确定是否使用该模型，进而可以减少资源开销的浪费。
第三方面，本申请实施例提供一种通信方法，包括：第一数据分析网元将模型的第一信息更新为所述模型的第二信息；所述第一数据分析网元确定所述模型的第二信息的索引信息，所述第二信息的索引信息包括第一标识信息，所述第一标识信息用于指示所述模型的第二信息；所述第一数据分析网元向所述第二数据分析网元发送所述第二信息的索引信息，所述第二信息的索引信息用于所述模型的第二信息的获取。其中，模型的第二信息的索引信息，也可以称为第二信息对应的模型索引信息。
基于上述方案，第一数据分析网元对模型进行更新，得到模型的第二信息后，可以将该第二信息的索引信息发送给第二数据分析网元，从而第二数据分析网元可以根据该索引信息获取到新的模型信息，即该第二信息，进而第二数据分析网元可以根据新的模型信息更新模型，实现模型性能提升。
在一种可能的实现方法中，所述第二信息的索引信息还包括以下一项或多项：所述模型对应的分析类型标识、所述模型的标识、所述模型的第二信息的版本信息。
在一种可能的实现方法中，所述第一数据分析网元从所述第二数据分析网元接收所述模型的第一信息的索引信息，所述第一信息的索引信息包括第二标识信息，所述第二标识信息用于指示所述模型的第一信息；所述第一数据分析网元根据所述第一信息的索引信息获取所述模型的第一信息。
在一种可能的实现方法中，所述第一信息的索引信息还包括以下一项或多项：所述模型对应的分析类型标识、所述模型的标识、所述模型的第一信息的版本信息。
在一种可能的实现方法中，第一数据分析网元将模型的第一信息更新为所述模型的第 二信息，包括：所述第一数据分析网元从所述第二数据分析网元获取第一请求，所述第一请求用于更新所述模型的第一信息，所述第一请求包括所述模型的第一信息的索引信息；所述第一数据分析网元根据所述第一信息的索引信息获取所述模型的第一信息；所述第一数据分析网元更新所述模型的第一信息，得到所述模型的第二信息。
在一种可能的实现方法中，第一数据分析网元从第二数据分析网元接收模型的第一信息的索引信息，包括：所述第一数据分析网元向所述第二数据分析网元发送第二请求，所述第二请求用于请求所述模型的第一信息的索引信息，所述第二请求包括所述模型对应的分析类型标识；所述第一数据分析网元从所述第二数据分析网元接收第二响应，所述第二响应包括所述模型的第一信息的索引信息。
在一种可能的实现方法中，所述第一数据分析网元通过网络存储网元从第二数据分析网元接收所述模型的第一信息的索引信息。
在一种可能的实现方法中，所述第一数据分析网元通过网络存储网元向第二数据分析网元发送所述模型的第二信息的索引信息。
在一种可能的实现方法中，所述第一数据分析网元为分布式学习中的客户端数据分析网元，所述第二数据分析网元为分布式学习中的服务端数据分析网元。
在一种可能的实现方法中，所述分布式学习为联邦学习。
在一种可能的实现方法中，所述第一数据分析网元为支持推理功能的数据分析网元，所述第二数据分析网元为支持训练功能的数据分析网元。
第四方面，本申请实施例提供一种通信装置，该装置可以是数据分析网元，还可以是用于数据分析网元的芯片。该装置具有实现上述第一方面至第三方面、或第一方面至第三方面的各可能的实现方法的功能。该功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能相对应的模块。
第五方面，本申请实施例提供一种通信装置，包括处理器和存储器；该存储器用于存储计算机执行指令，当该装置运行时，该处理器执行该存储器存储的该计算机执行指令，以使该装置执行如上述第一方面至第三方面的方法及第一方面至第三方面的各可能的实现方法中的任意方法。
第六方面，本申请实施例提供一种通信装置，包括用于执行上述第一方面至第三方面的方法及第一方面至第三方面的各可能的实现方法中的任意方法的各个步骤的单元或手段(means)。
第七方面，本申请实施例提供一种通信装置，包括处理器和接口电路，所述处理器用于通过接口电路与其它装置通信，并执行上述第一方面至第三方面的方法及第一方面至第三方面的各可能的实现方法中的任意方法。该处理器包括一个或多个。
第八方面，本申请实施例提供一种通信装置，包括处理器，用于与存储器相连，用于调用所述存储器中存储的程序，以执行上述第一方面至第三方面的方法及第一方面至第三方面的各可能的实现方法中的任意方法。该存储器可以位于该装置之内，也可以位于该装置之外。且该处理器包括一个或多个。
第九方面，本申请实施例还提供一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，当其在计算机上运行时，使得处理器执行上述第一方面至第三方面的方法及第一方面至第三方面的各可能的实现方法中的任意方法。
第十方面，本申请实施例还提供一种计算机程序产品，该计算机产品包括计算机程序， 当计算机程序运行时，使得上述第一方面至第三方面的方法及第一方面至第三方面的各可能的实现方法中的任意方法。
第十一方面，本申请实施例还提供一种芯片系统，包括：处理器，用于执行上述第一方面至第三方面的方法及第一方面至第三方面的各可能的实现方法中的任意方法。
第十二方面，本申请实施例还提供一种通信系统，包括：用于执行上述第一方面或第一方面的任意实现方法的第一数据分析网元，和用于执行上述第二方面或第二方面的任意实现方法的第二数据分析网元。
附图说明
图1为5G网络架构示意图；
图2为5G网络中的NF注册/发现/更新流程示意图；
图3为训练和推断分离架构下的训练NWDAF和推断NWDAF的工作流程示意图；
图4为本申请实施例适用的网络架构示意图；
图5至图12为本申请实施例提供的训练-推断分离场景下保证模型有效性的八种方法示意图；
图13为本申请实施例提供的一种通信方法示意图；
图14(a)为横向联邦学习的训练过程；
图14(b)为本申请实施例提供的又一种通信方法示意图；
图15为本申请实施例提供的一种通信装置示意图；
图16为本申请实施例提供的又一种通信装置示意图。
具体实施方式
为了使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请作进一步地详细描述。方法实施例中的具体操作方法也可以应用于装置实施例或系统实施例中。其中，在本申请的描述中，除非另有说明，“多个”的含义是两个或两个以上。
无线机器学习模型驱动网络架构(wireless Machine Learning-based Network，wMLN)主要解决机器学习模型在无线网络中的生命周期管理问题。该网络架构中的模型训练功能和模型推断功能是与机器学习模型紧密相关的两个核心功能模块。模型训练功能对计算能力要求较高，且需要较大的数据量，通常需要部署在算力和数据强大的集中网元。考虑到推断实时性等要求，模型推断功能通常部署在靠近业务功能的本地网元中，以减少传输和处理时延。因此，模型训练功能和推断功能分离是一种典型的部署场景。
使能网络自动化(enabler of Network Automation，eNA)架构是一个基于网络数据分析功能(Network Data Analytics Function，NWDAF)的智能网络架构。如图1所示，NWDAF是第三代合作伙伴计划(3rd generation partnership project，3GPP)引入的标准化网元，主要可以用于收集网络数据(包括终端数据、基站数据、传输网数据、核心网数据、网管数据以及第三方应用数据中的一种或者多种)，并提供网络数据分析服务，可以输出数据分析结果，供网络、网管及应用执行策略决策使用。NWDAF可以利用机器学习模型进行数据分析。3GPP Release 17中NWDAF的功能被分解，包括数据收集功能、模型训练功能以及模型推断功能。在训练功能和推断功能分离的场景下，同一模型的训练功能和推断功能分开部署在不同NWDAF实例中，部署训练功能的NWDAF(简称为训练NWDAF)可以 提供训练后的模型，部署推断功能的NWDAF(简称为推断NWDAF)通过获取训练NWDAF提供的模型进行模型推断，提供数据分析服务。
机器学习模型的训练通常是通过学习某一组输入特征与输出目标之间的映射来进行的，通过优化某些损失函数来使得机器学习模型的输出结果(即预测值)与实际结果(即标签值/真实值)之间的误差最小化。当训练出最优模型后，利用该模型的输出对未来的情况进行预测。在理想情况下，假定未来将使用到的数据与模型训练期间所使用的数据类似，具体来说，可能假设训练时的输入特征和预测时的输入特征的分布保持恒定。但是实际中这种假设通常不成立，数据的特征会随着时间的推移由于网络部署变化、应用层业务要求变化、网络实际用户分布变化等而变化，因此，模型的性能(即泛化能力)会随着时间推移逐渐下降。具体的表现可能是，模型的准确率降低，即模型的预测值与真实值之间的误差变大。
图1所示的5G网络架构中可包括三部分，分别是终端设备部分、数据网络(data network，DN)和运营商网络部分。下面对其中的部分网元的功能进行简单介绍说明。
其中，运营商网络可包括以下网元中的一个或多个：鉴权服务器功能(Authentication Server Function，AUSF)网元、网络开放功能(network exposure function，NEF)网元、策略控制功能(Policy Control Function，PCF)网元、统一数据管理(unified data management，UDM)、统一数据库(Unified Data Repository，UDR)、网络存储功能(Network Repository Function，NRF)网元、应用功能(Application Function，AF)网元、接入与移动性管理功能(Access and Mobility Management Function，AMF)网元、会话管理功能(session management function，SMF)网元、RAN以及用户面功能(user plane function，UPF)网元、NWDAF网元等。上述运营商网络中，除无线接入网部分之外的部分可以称为核心网络部分。
在具体实现中，本申请实施例中的终端设备，可以是用于实现无线通信功能的设备。其中，终端设备可以是5G网络或者未来演进的公共陆地移动网络(public land mobile network，PLMN)中的用户设备(user equipment，UE)、接入终端、终端单元、终端站、移动站、移动台、远方站、远程终端、移动设备、无线通信设备、终端代理或终端装置等。接入终端可以是蜂窝电话、无绳电话、会话启动协议(session initiation protocol，SIP)电话、无线本地环路(wireless local loop，WLL)站、个人数字助理(personal digital assistant，PDA)、具有无线通信功能的手持设备、计算设备或连接到无线调制解调器的其它处理设备、车载设备或可穿戴设备，虚拟现实(virtual reality，VR)终端设备、增强现实(augmented reality，AR)终端设备、工业控制(industrial control)中的无线终端、无人驾驶(self driving)中的无线终端、远程医疗(remote medical)中的无线终端、智能电网(smart grid)中的无线终端、运输安全(transportation safety)中的无线终端、智慧城市(smart city)中的无线终端、智慧家庭(smart home)中的无线终端等。终端可以是移动的，也可以是固定的。
上述终端设备可通过运营商网络提供的接口(例如N1等)与运营商网络建立连接，使用运营商网络提供的数据和/或语音等服务。终端设备还可通过运营商网络访问DN，使用DN上部署的运营商业务，和/或第三方提供的业务。其中，上述第三方可为运营商网络和终端设备之外的服务方，可为终端设备提供其他数据和/或语音等服务。其中，上述第三方的具体表现形式，具体可根据实际应用场景确定，在此不做限制。
RAN作为接入网网元是运营商网络的子网络，是运营商网络中业务节点与终端设备之间的实施系统。终端设备要接入运营商网络，首先是经过RAN，进而可通过RAN与运营商网络的业务节点连接。本申请中的RAN设备，是一种为终端设备提供无线通信功能的设备，RAN设备也称为接入网设备。本申请中的RAN设备包括但不限于：5G中的下一代基站(g nodeB，gNB)、演进型节点B(evolved node B，eNB)、无线网络控制器(radio network controller，RNC)、节点B(node B，NB)、基站控制器(base station controller，BSC)、基站收发台(base transceiver station，BTS)、家庭基站(例如，home evolved nodeB，或home node B，HNB)、基带单元(baseBand unit，BBU)、传输点(transmitting and receiving point，TRP)、发射点(transmitting point，TP)、移动交换中心等。
AMF网元，主要进行移动性管理、接入鉴权/授权等功能。此外，还负责在UE与PCF间传递用户策略。
SMF网元，主要进行会话管理、PCF下发控制策略的执行、UPF的选择、UE互联网协议(internet protocol，IP)地址分配等功能。
UPF网元，作为和数据网络的接口UPF，完成用户面数据转发、基于会话/流级的计费统计，带宽限制等功能。
UDM网元，主要负责管理签约数据、用户接入授权等功能。
UDR，主要负责签约数据、策略数据、应用数据等类型数据的存取功能。
NEF网元，主要用于支持能力和事件的开放。
AF网元，主要传递应用侧对网络侧的需求，例如，服务质量(Quality of Service，QoS)需求或用户状态事件订阅等。AF可以是第三方功能实体，也可以是运营商部署的应用服务，如IP多媒体子系统(IP Multimedia Subsystem，IMS)语音呼叫业务。
PCF网元，主要负责针对会话、业务流级别进行计费、QoS带宽保障及移动性管理、UE策略决策等策略控制功能。
NRF网元，可用于提供网元发现功能，基于其他网元的请求，提供网元类型对应的网元信息。NRF还提供网元管理服务，如网元注册、更新、去注册以及网元状态订阅和推送等。
AUSF网元：主要负责对用户进行鉴权，以确定是否允许用户或设备接入网络。
DN，是位于运营商网络之外的网络，运营商网络可以接入多个DN，DN上可部署多种业务，可为终端设备提供数据和/或语音等服务。例如，DN是某智能工厂的私有网络，智能工厂安装在车间的传感器可为终端设备，DN中部署了传感器的控制服务器，控制服务器可为传感器提供服务。传感器可与控制服务器通信，获取控制服务器的指令，根据指令将采集的传感器数据传送给控制服务器等。又例如，DN是某公司的内部办公网络，该公司员工的手机或者电脑可为终端设备，员工的手机或者电脑可以访问公司内部办公网络上的信息、数据资源等。
图1中Nnwdaf、Nausf、Nnef、Npcf、Nudm、Naf、Namf、Nsmf、N1、N2、N3、N4，以及N6为接口序列号。这些接口序列号的含义可参见3GPP标准协议中定义的含义，在此不做限制。
需要说明的是，本申请实施例中，数据分析网元可以是图1所示的NWDAF网元，也可以是未来通信系统中具有上述NWDAF网元的功能的其它网元。网络存储网元可以是图1所示的NRF网元，也可以是未来通信系统中具有上述NRF网元的功能的其它网元。为 便于说明，本申请实施例中，以数据分析网元为NWDAF网元，网络存储网元为NRF网元为例进行说明。并且，将NWDAF网元进一步划分为训练NWDAF网元和推断NWDAF网元。
如图2所示，为5G网络中的NF注册/发现/更新流程示意图。5G网络中的NRF主要用于网络功能(Network Function，NF)的管理，这里的网络功能比如可以是SMF、AMF、NEF、AUSF、NWDAF、PCF等等。NRF支持的功能包括：
1)NF注册/更新/去注册：可用的NF实例(NF instance)将自身可提供的服务注册在NRF中，注册信息通过NF配置文件(NF profile)描述，NF配置文件包括NF类型，NF服务名称，NF地址等信息。NRF负责维护这些NF配置文件。当NF需要更新或删除时，NRF对NF配置文件进行相应的修改和删除。
2)NF发现：NRF接收来自NF实例的NF发现请求，并将发现的NF实例信息提供给请求的NF实例。例如，AMF向NRF请求发现SMF实例。再比如，某个AMF向NRF请求发现另一个AMF实例。
3)NF状态通知：NRF向订阅的NF服务消费者通知新注册/更新/注销的NF实例和其提供的NF服务。
在图2中，NF注册过程包括步骤201至步骤203。
步骤201，NF1向NRF发送NF注册请求，携带NF配置文件。
该NF配置文件包括NF类型，NF服务名称，NF地址等信息。
步骤202，NRF存储该NF配置文件。
步骤203，NRF向NF1发送NF注册响应。
该NF注册响应用于通知NF注册成功。
NF发现过程包括步骤204至步骤205。
步骤204，NF2向NRF发送NF发现请求消息，携带需要查找的NF的条件信息，如NF类型(NF type)。
步骤205，NRF向NF2发送NF发现响应，携带符合条件的NF实例信息，如NF标识(NF ID)或NF IP地址。
NF更新过程包括步骤206a至步骤210。
步骤206a，NF2向NRF发送NF状态订阅请求，携带NF实例，用于请求订阅该NF实例的状态信息。
当NF2向NRF订阅了某个NF实例的状态信息(以下以订阅NF1的状态信息为例)之后，后续NRF发现该NF实例的状态信息发生改变，则NRF会向NF2发送该NF实例的更新的状态信息。
步骤206b，NRF向NF2发送NF状态订阅响应。
该NF状态订阅响应用于通知NF状态订阅成功。
步骤207，NF1向NRF发送NF更新请求，携带更新的NF配置文件。
步骤208，NRF更新NF配置文件。
也即，NRF根据接收到的更新的NF配置文件，对存储的NF配置文件进行更新。
步骤209，NRF向NF1发送NF更新响应。
NF更新响应用于指示NF配置文件更新成功。
步骤210，NRF向NF2发送NF状态改变通知，携带更新的NF配置文件。
也即，NRF向之前订阅过NF1的状态信息的NF2发送给NF状态改变通知。
基于上述过程，可以结合NRF，实现NF的注册，发现以及更新的功能。
需要说明的是，上述NF注册、NF发现、NF更新的过程并不一定是连续发生的，这里只是给出一个流程示例，说明通常发生的先后顺序。
如图3所示，为训练和推断分离架构下的训练NWDAF和推断NWDAF的工作流程示意图。其中，各个网元功能介绍如下：
NRF：负责NF管理，提供的接口服务包括NF注册/去注册/更新，NF状态订阅/通知等。
训练NWDAF：负责模型训练，训练好的模型可被其它NWDAF(如推断NWDAF)使用。
推断NWDAF：负责模型推断，利用推断结果进行数据分析，并输出数据分析结果。
NF：负责某个特定业务功能，可以调用推断NWDAF的服务获取数据分析结果。
该图3所示的流程包括以下步骤：
步骤301，训练NWDAF向NRF发送NF注册请求，携带NF配置文件。
该NF配置文件包括NF类型，NF服务名称(NF Service)，分析类型标识(Analytics ID)等信息。
其中，NF类型可以是NWDAF。
NF服务名称可以是提供模型服务(ModelProvision)。
分析类型标识用于指示训练NWDAF提供的某种特定分析类型，比如可以是Service Experience，Network Performance，UE Mobility等。
步骤302，NRF存储该NF配置文件。
步骤303，NRF向训练NWDAF发送NF注册响应。
该NF注册响应用于通知训练NWDAF注册成功。
步骤304，推断NWDAF向NRF发送NF发现请求，携带NF配置文件。
比如携带的NF配置文件包含NF类型(如NWDAF)，NF服务名称(如ModelProvision)，以及携带Analytics ID，则该NF发现请求用于请求从NRF获取一个对应该Analytics ID的训练NWDAF。
步骤305，NRF向推断NWDAF发送NF发现响应，其中携带NWDAF实例。
其中，携带的NWDAF实例是训练NWDAF的一个实例，可以用训练NWDAF的ID或IP地址来表示。
其中，上述步骤301-步骤305可选步骤。例如，如果推断NWDAF上配置了训练NWDAF的NF配置信息，步骤301-步骤305可以不执行。
步骤306，推断NWDAF向训练NWDAF发送模型请求，其中携带Analytics ID。
其中，推断NWDAF可以基于从NRF获取到的训练NWDAF的ID或IP地址，向训练NWDAF发送模型请求，携带的Analytics ID用于指示请求获取与该Analytics ID对应的模型。
步骤307，训练NWDAF向推断NWDAF发送模型响应，携带模型信息。
其中，模型(也称为机器学习模型，Machine Learning Model，ML Model)信息用于 描述根据样本输入数据确定样本输出数据的方法，模型信息中可以包括但不限制于以下信息中的一个或者多个：输入数据对应的特征类型、输入数据对应的特征类型的特征提取方法(函数关系)、输出数据对应的类型(类别标签、连续数值等)、模型使用的算法类型、模型的类别(分类、回归、聚类等)、模型的参数。以猫-狗分类模型为例，该模型可以根据未知动物的形体样本输入数据确定该样本是猫还是狗，这其中，输入数据的特征类型可以是动物体重、毛长、叫声，输入数据对应的特征类型动物体重的提取方法可以是最大最小归一化，输出数据对应的类型为猫或者狗，模型使用的算法类型可以为深度神经网络(deep neural network，DNN)，模型的类别为分类，模型的参数包括但不限于：神经网络的层数、每一层使用的激活函数、每一层激活函数对应的一个或者多个函数参数值。值得说明的是，本发明中所有涉及模型信息(如第一模型信息、第二模型信息等)、模型的信息(如模型的第一信息、模型的第二信息等)都可以参考关于模型信息的描述，其他地方不再赘述。
上述步骤301至步骤307是训练NWDAF提供模型服务的流程。基于该流程，训练NWDAF将NF配置文件注册至NRF，后续推断NWDAF可以从NRF获取到训练NWDAF实例，进而推断NWDAF可以向训练NWDAF请求获取特定类型的模型信息。也即，训练NWDAF可以向推断NWDAF提供模型服务。
步骤308，推断NWDAF向NRF发送NF注册请求，携带NF配置文件。
该NF配置文件包括NF类型，NF服务名称(NF Service)，分析类型标识(Analytics ID)等信息。
其中，NF类型可以是NWDAF。
NF服务名称可以是提供分析服务(Analytics)。
分析类型标识用于指示训练NWDAF提供的某种特定分析类型，比如可以是Service Experience，Network Performance，UE Mobility等。
步骤309，NRF存储该NF配置文件。
步骤310，NRF向推断NWDAF发送NF注册响应。
该NF注册响应用于通知推断NWDAF注册成功。
步骤311，NF向NRF发送NF发现请求，携带NF配置文件。
该NF指的是一个NF消费者(NF consumer)，比如可以是SMF、AMF或UPF等等。
比如携带的NF配置文件包含NF类型(如NWDAF)，NF服务名称(如ModelProvision)，以及携带Analytics ID，则该NF发现请求用于请求从NRF获取对应该Analytics ID的推断NWDAF。
步骤312，NRF向NF发送NF发现响应，其中携带NWDAF实例。
其中，携带的NWDAF实例是推断NWDAF的一个实例，可以用推断NWDAF的ID或IP地址来表示。
值得说明的是，上述步骤308-步骤312可选执行。例如，如果NF上配置了推断NWDAF的NF配置信息，步骤308-步骤312可以不执行。
步骤313，NF向推断NWDAF发送分析订阅，其中携带Analytics ID。
其中，NF可以基于从NRF获取到的推断NWDAF的ID或IP地址，向推断NWDAF发送分析订阅，携带的Analytics ID用于指示订阅获取与该Analytics ID对应的数据分析结果。
步骤314，推断NWDAF向NF发送分析结果通知，携带数据分析结果。
上述步骤308至步骤314是推断NWDAF提供分析服务的流程。基于该流程，推断NWDAF将NF配置文件注册至NRF，后续NF可以从NRF获取到推断NWDAF实例，进而NF可以向推断NWDAF请求获取特定类型的数据分析结果。也即，推断NWDAF可以向NF提供数据分析服务。
作为一种可替代的实现方法，上述步骤313至步骤314可以使用以下步骤313’至步骤314’替换：
步骤313’，NF向推断NWDAF发送分析请求，其中携带Analytics ID。
其中，NF可以基于从NRF获取到的推断NWDAF的ID或IP地址，向推断NWDAF发送分析订阅，携带的Analytics ID用于指示请求获取与该Analytics ID对应的数据分析结果。
步骤314’，推断NWDAF向NF发送分析结果响应，携带数据分析结果。
该步骤313’至步骤314’是每次需要主动发送分析请求，推断NWDAF才向NF发送数据分析结果，而上述步骤313至步骤314是只需要订阅一次，后续推断NWDAF在产生新的数据分析结果时主动向NF发送数据分析结果。
上述图3所示的模型训练与模型使用过程所存在的问题是：随着时间推移，推断NWDAF本地可以根据推断数据确定推断结果，然后根据推断数据的真实结果以及推断结果确定模型的使用效果(也就是模型性能评估结果)，推断NWDAF根据使用效果确定机器学习模型性能下降，但是训练-推断分离的场景下训练NWDAF无法感知推断NWDAF中的模型使用效果，且推断NWDAF没有能力进行模型训练，因此，现有技术中无法实现在模型性能下降时进行再训练和模型更新，也就无法保证模型在运行过程中性能一直良好。如果推断NWDAF继续使用性能下降的模型进行数据分析，可能导致数据分析结果不准确，影响模型性能。
为解决上述问题，本申请实施例提出建立一种模型性能监控和反馈机制，对推断NWDAF中运行的模型性能进行评估，当模型性能下降至一定程度时训练NWDAF可以感知并及时进行再训练，推断NWDAF可利用再训练得到的性能良好的新模型进行模型更新(或者替换)，保证模型的使用效果。其中，监控、反馈、再训练和更新机制可以通过NRF实现，也可以由训练NWDAF与推断NWDAF直接交互实现。
本申请实施例应用的系统架构是eNA架构，具体地，本申请实施例针对的是模型训练和推断功能分离部署的场景，即训练功能和推断功能部署在不同的NWDAF实例中。如图4所示，为本申请实施例适用的一种网络架构示意图。训练NWDAF，推断NWDAF和NF，都需要通过Nnrf接口服务在NRF中进行注册。推断NWDAF通过Nnwdaf接口服务向训练NWDAF请求模型，NF通过Nnwdaf接口服务向推断训练NWDAF请求数据分析结果。
下面对本申请实施例提供的方案进行说明。
实施例一
如图5所示，为本申请实施例提供的一种训练-推断分离场景下保证模型有效性的方法流程示意图。
该实施例一考虑通过NRF更新注册信息，实现性能监控和模型更新。主要涉及：
1、模型性能监控与反馈：推断NWDAF在NRF处的注册信息增加模型状态信息，推 断NWDAF进行模型性能监控，当判断模型需要再训练时通过NRF更新模型状态信息，NRF通知训练NWDAF模型状态更新，触发训练NWDAF对模型进行再训练。
2、模型更新：训练NWDAF在NRF处的注册信息增加模型索引信息，训练NWDAF再训练得到新模型后通过NRF更新模型索引信息，NRF通知推断NWDAF有新模型可以使用，推断NWDAF主动向训练NWDAF请求新模型并完成模型更新。
该实施例包括以下步骤：
步骤501，训练NWDAF向NRF注册。
训练NWDAF向NRF发送NF注册请求，携带NF配置文件，该NF配置文件包括NF类型，NF服务名称(例如，NF Service)，分析类型标识(例如，Analytics ID)等信息，以及还包括模型索引信息。该模型索引信息可以是模型版本号(例如，version)、位置信息(例如，location)或统一资源定位符(Uniform Resource Locator，URL)等。其中，version表示模型版本，location或URL表示模型的存储位置，三者中使用任意一个都可以。可选的，当模型索引信息是位置信息或URL，该位置信息或URL中也可以包含模型版本。可选的，位置信息可以是IP地址。
相应地，NRF存储NF配置文件以及向训练NWDAF发送NF注册响应。
步骤502，推断NWDAF向NRF注册。
推断NWDAF向NRF发送NF注册请求，携带NF配置文件，该NF配置文件包括NF类型，NF服务名称(例如，NF Service)，分析类型标识(例如，Analytics ID)等信息，以及还包括模型状态信息。该模型状态信息用于指示模型使用状态。
该模型状态信息的可选取值包括但不限于：
1)空‘null’：表示没有模型可以使用。
2)允许‘ok’：表示模型性能良好，可以对外提供分析服务。
3)限定‘limited’：表示模型性能有所下降，但仍可以提供服务，需要进行再训练。
4)停止‘stopped’：表示模型关闭，已停止提供服务。
其中，在注册过程中，NF注册请求携带的模型状态信息为‘null’，即推断NWDAF上当前没有模型可以使用。
相应地，NRF存储NF配置文件以及向推断NWDAF发送NF注册响应。
步骤503，推断NWDAF发现训练NWDAF，并向训练NWDAF请求获取模型信息。
该过程可以参考图3实施例中的步骤304至步骤307，不再赘述。基于该过程，推断NWDAF可以从训练NWDAF中获取到模型信息。
步骤504，推断NWDAF向NRF订阅训练NWDAF的状态。
后续训练NWDAF在NRF中注册的NF配置文件发生更新时，NRF通知推断NWDAF。
步骤505，训练NWDAF向NRF订阅推断NWDAF的状态。
后续推断NWDAF在NRF中注册的NF配置文件发生更新时，NRF通知训练NWDAF。
上述步骤504与步骤505相互之间没有固定的先后顺序。
步骤506，推断NWDAF向NRF发送NF更新请求，携带更新的NF配置文件。
该更新的NF配置文件至少携带更新的模型状态信息，该更新的模型状态信息比如可以是“ok”。可选的，该更新的NF配置文件还携带Analytics ID，用于标识要更新的模型。可选的，该更新的NF配置文件还携带NF类型，NF服务名称(NF Service)等。
步骤507，NRF更新NF配置文件。
也即，NRF根据接收到的更新的NF配置文件，对存储的NF配置文件进行更新。
步骤508，NRF向推断NWDAF发送NF更新响应。
该NF更新响应用于通知推断NWDAF的NF配置文件更新成功。
步骤509，NRF向训练NWDAF发送NF状态更新通知，其中携带更新的模型状态信息。
该更新的模型状态信息比如可以是‘ok’。
可选的，该NF状态更新通知还携带指示信息，用于指示更新的类型是模型状态信息更新。
由于上述步骤505中，训练NWDAF向NRF订阅了推断NWDAF的状态，因此当NRF中存储的推断NWDAF的NF配置文件发生更新后，NRF通知训练NWDAF。
步骤510，推断NWDAF确定模型需要再训练。
判断依据可以是对模型性能的评估结果不满足模型性能要求(比如模型精度下降到80％以下，其中80％为模型精度要求)，也可以是NF上报的业务关键性能指标(Key Performance Indicator，KPI)不满足KPI要求(比如KPI下降到KPI要求以下)。本发明其他实施例中涉及判定模型是否需要再训练或者需要更新的方法都可以参考此处描述，不再赘述。
需要说明的是，该步骤510发生在推断NWDAF中的模型运行过程中，出现的时间不固定。
步骤511，推断NWDAF向NRF发送NF更新请求，携带更新的NF配置文件。
该更新的NF配置文件至少携带更新的模型状态信息，该更新的模型状态信息比如可以是‘limited’。可选的，该更新的NF配置文件还携带Analytics ID，用于标识要更新的模型。可选的，该更新的NF配置文件还携带NF类型，NF服务名称(NF Service)等。
NRF更新存储的NF配置文件，然后向推断NWDAF发送NF更新响应。
步骤512，NRF向训练NWDAF发送NF状态更新通知，其中携带更新的模型状态信息。
该更新的模型状态信息可以是‘limited’。
可选的，该NF状态更新通知还携带指示信息，用于指示更新的类型是模型状态信息更新。
由于上述步骤505中，训练NWDAF向NRF订阅了推断NWDAF的状态，因此当NRF中存储的推断NWDAF的NF配置文件发生更新后，NRF通知训练NWDAF。
步骤513，训练NWDAF对模型启动再训练。
训练NWDAF对模型启动再训练，得到训练后的模型以及相应的模型索引信息，如模型版本号、位置信息或URL等。
步骤514，训练NWDAF向NRF发送NF更新请求，携带更新的NF配置文件。
该更新的NF配置文件至少携带更新的模型索引信息，该更新的模型索引信息比如可以是更新的模型版本信息、更新的模型位置信息或更新的模型URL等。可选的，该更新的NF配置文件还携带Analytics ID，用于标识要更新的模型。可选的，该更新的NF配置文件还携带NF类型，NF服务名称(NF Service)等。
NRF更新存储的NF配置文件，然后向训练NWDAF发送NF更新响应。
步骤515，NRF向推断NWDAF发送NF状态更新通知，其中携带更新的模型索引信息。
该更新的模型索引信息比如可以是更新的模型版本信息、更新的模型位置信息或更新的模型URL等。
可选的，该NF状态更新通知还携带指示信息，用于指示更新的类型是模型索引信息更新。
由于上述步骤504中，推断NWDAF向NRF订阅了训练NWDAF的状态，因此当NRF中存储的训练NWDAF的NF配置文件发生更新后，NRF通知推断NWDAF。
步骤516，推断NWDAF向训练NWDAF发送模型请求，携带Analytics ID和更新的模型索引信息。
Analytics ID用于指示与该Analytics ID对应的模型。
步骤517，训练NWDAF向推断NWDAF发送模型响应，携带模型信息。
该模型信息包括与更新的模型索引信息对应的模型信息，也即获取到新的模型对应的模型信息。
可选的，模型响应中携带的模型信息可以是新模型的参数项取值、或新的模型(如模型文件或包含模型的镜像文件)、或新的模型地址(如URL或IP地址)。
其中，模型文件是使用第三方框架保存下来的模型持久化文件，如使用人工智能框架TensorFlow保存的.pb格式的模型文件。模型镜像文件是一个包含模型的镜像软件包，其中可以包含模型文件以及与模型使用相关的其他多个文件。
需要说明的是，如果步骤516中携带的模型索引信息是新的模型地址，则推断NWDAF可以直接根据该地址信息进一步获取新的模型信息，无需执行步骤516-517。比如，推断NWDAF根据URL通过文件传输协议(File Transfer Protocol，FTP)获取包含新的模型信息的文件(如包含新模型的参数项取值的文件、或新的模型文件、或包含新模型的镜像文件)。
如果步骤516中携带的模型索引信息是模型版本号，则模型响应中携带的模型信息可以是新模型的参数项取值、或新的模型(模型文件或包含模型的镜像)，也可以是新模型的地址(如URL或IP地址)。如果模型响应中携带的模型信息是新模型的地址(如URL或IP地址)，推断NWDAF可以根据该地址信息进一步获取新的模型信息。
步骤518，推断NWDAF进行模型更新。
也即，推断NWDAF根据接收到的新的模型信息，对正在使用的旧模型进行更新或者替换。
可选的，推断NWDAF在模型更新前，对新的模型信息进行本地测试，测试通过后再进行更新或者替换。
步骤519，推断NWDAF向NRF发送NF更新请求，携带更新的NF配置文件。
该更新的NF配置文件至少携带更新的模型状态信息，该更新的模型状态信息比如可以是’ok’。可选的，该更新的NF配置文件还携带Analytics ID，用于标识要更新的模型。可选的，该更新的NF配置文件还携带NF类型，NF服务名称(NF Service)等。
NRF更新存储的NF配置文件，然后向推断NWDAF发送NF更新响应。
步骤520，NRF向训练NWDAF发送NF状态更新通知，其中携带更新的模型状态信息。
该更新的模型状态信息可以是‘ok’。
可选的，该NF状态更新通知还携带指示信息，用于指示更新的类型是模型状态信息 更新。
其中，上述步骤511-步骤520可选步骤。例如，如果步骤510中推断NWDAF确定模型不需要再训练或者推断NWDAF可以容忍模型性能下降到模型性能要求以下，步骤511-步骤520可以不执行。
由于上述步骤505中，训练NWDAF向NRF订阅了推断NWDAF的状态，因此当NRF中存储的推断NWDAF的NF配置文件发生更新后，NRF通知训练NWDAF。
基于上述实施例，一方面，当推断NWDAF中使用的模型的性能下降时，可以通过NRF通知训练NWDAF进行模型再训练，训练结束后推断NWDAF可利用新模型更新或者替换旧模型，保证模型的使用效果。
另一方面，当推断NWDAF为多个时，如果只是其中一个推断NWDAF X的性能下降，并且通知训练NWDAF进行再训练，训练结束后，除了推断NWDAF X可以利用新模型更新或者替换旧模型，多个推断NWDAF中的其他任何一个推断NWDAF也可以利用上述机制获取新模型，并且利用新模型更新或者替换旧模型，这样可以保证多个推断NWDAF上模型的使用效果。针对多推断NWDAF的场景，详细过程的示例参见实施例二。
实施例二
如图6所示，为本申请实施例提供的又一种训练-推断分离场景下保证模型有效性的方法流程示意图。
该实施例二，是在上述实施例一的基础上，考虑有多个推断NWDAF存在的场景，以下以有两个推断NWDAF(分别用推断NWDAF1和推断NWDAF2来表示)为例，推断NWDAF1和推断NWDAF2中使用相同的模型进行数据分析，该模型都来自同一训练NWDAF。某一时刻，NWDAF1中模型性能下降需要再训练，推断NWDAF2中模型不需要再训练。这里仅以推断NWDAF2为例表示存在除了请求再训练的推断NWDAF1外，还存在其他使用相同模型的推断NWDAF。
由于推断NWDAF2也订阅了训练NWDAF状态，因此，当训练NWDAF再训练得到新模型之后，推断NWDAF2也会收到来自NRF的通知。一方面，如果新模型比推断NWDAF2中模型效果更好，推断NWDAF2可以利用新模型进一步改善数据分析效果，另一方面，由于推断NWDAF2中模型此时并不是必须要更新，推断NWDAF2需要先获取新模型并进行本地评估后才可以确定是否需要更新。如果获取新模型后推断NWDAF2最终决定不更新，则浪费了一部分传输资源。本实施例考虑在训练NWDAF的注册信息进一步增加模型性能信息，包括精度、所需计算量等，帮助其它暂不需要更新的推断NWDAF判断是否需要请求新模型。
该实施例包括以下步骤：
步骤601，训练NWDAF向NRF注册。
训练NWDAF向NRF发送NF注册请求，携带NF配置文件，该NF配置文件包括NF类型，NF服务名称(NF Service)，分析类型标识(Analytics ID)等信息，以及还包括模型索引信息和模型性能信息。该模型索引信息可以是模型版本号(version)、位置信息(location)或URL等。其中，version表示模型版本，location或URL表示模型的存储位置，三者中使用任意一个都可以。可选的，当模型索引信息是location或URL，该location或URL中也可以包含version。该模型性能信息用于指示模型的性能，比如可以包括模型 准确度、达到该准确度所需的硬件能力信息、模型推理所需计算量、模型的推理时长、模型的大小等。
可选的，NF配置文件还可以包含模型使用的算法、人工智能框架、模型的输入特征等信息。
相应地，NRF存储NF配置文件以及向训练NWDAF发送NF注册响应。
步骤602至步骤613，类似于上述实施例一中的步骤502至步骤513。
需要说明的是，该步骤602至步骤613中，涉及推断NWDAF1的相关操作以及推断NWDAF2的相关操作，均可以分别参考上述步骤502至步骤513中关于推断NWDAF的相关操作。并且，在步骤610至步骤611(参考步骤510至步骤611)中，是推断NWDAF1确定模型需要再训练，然后向NRF发送NF更新请求，从而触发训练NWDAF对模型启动再训练。
步骤614，训练NWDAF向NRF发送NF更新请求，携带更新的NF配置文件。
该更新的NF配置文件至少携带更新的模型索引信息以及更新的模型性能信息，该更新的模型索引信息比如可以是更新的模型版本信息、更新的模型位置信息或更新的模型URL等。可选的，该更新的NF配置文件还携带Analytics ID，用于标识要更新的模型。该更新的模型性能信息比如可以包括模型准确度、达到该准确度所需的硬件能力信息、模型推理所需计算量、模型的推理时长、模型的大小等。
可选的，该更新的NF配置文件还携带NF类型，NF服务名称(NF Service)等。
NRF更新存储的NF配置文件，然后向训练NWDAF发送NF更新响应。
步骤615，NRF分别向推断NWDAF1和推断NWDAF2发送NF状态更新通知，其中携带更新的模型索引信息和更新的模型性能信息。
该更新的模型索引信息比如可以是更新的模型版本信息、更新的模型位置信息或更新的模型URL等。
该更新的模型性能信息比如可以包括模型准确度、达到该准确度所需的硬件能力信息、模型推理所需计算量、模型的推理时长、模型的大小等。
可选的，该NF状态更新通知还携带指示信息，用于指示更新的类型是模型索引信息更新以及模型性能信息更新。
由于上述步骤中，推断NWDAF1和推断NWDAF2分别向NRF订阅了训练NWDAF的状态，因此当NRF中存储的训练NWDAF的NF配置文件发生更新后，NRF通知推断NWDAF1和推断NWDAF2。
步骤616，推断NWDAF2判断是否需要更新模型。
由于推断NWDAF2不是模型训练的触发者，因此当推断NWDAF2收到更新的模型索引信息后，需要判断是否需要更新模型。
可选的，推断NWDAF2可以根据自身算力、模型性能要求以及接收到的更新的模型性能信息，判断是否需要更新模型。或者，推断NWDAF2还可以根据正在使用的模型性能状况以及接收到的更新的模型性能信息，判断是否需要更新模型。
步骤617，推断NWDAF1向训练NWDAF发送模型请求，携带Analytics ID和更新的模型索引信息。
Analytics ID用于指示与该Analytics ID对应的模型。
由于推断NWDAF1是模型训练的触发者，因此当推断NWDAF1收到更新的模型索引 信息后，需要更新模型。
步骤618，训练NWDAF向推断NWDAF1发送模型响应，携带模型信息。
该模型信息包括与更新后的模型标识对应的模型信息，也即获取到新的模型对应的模型信息。
其中，模型信息的具体实现，可以参考前述实施例的描述。
步骤619，推断NWDAF1进行模型更新。
也即，推断NWDAF根据接收到的新的模型信息，对正在使用的旧模型进行更新或者替换。
可选的，推断NWDAF在模型更新前，对新的模型信息进行本地测试，测试通过后再进行更新或者替换。
需要说明的是，如果上述步骤616中，推断NWDAF2确定需要更新模型，则推断NWDAF2还需要执行类似于上述步骤617至步骤619的操作过程，以向训练NWDAF请求获取到更新的模型信息，然后根据接收到的新的模型信息，对正在使用的旧模型进行更新。如果上述步骤616中，推断NWDAF2确定不需要更新模型，则无需执行该模型更新流程。
基于上述实施例，订阅相同模型的推断NWDAF可以根据模型性能信息，确定是否需要请求新模型，以避免不必要的模型传输和本地评估过程，从而可以提升模型更新流程的效率以及节约资源。
实施例三
如图7所示，为本申请实施例提供的又一种训练-推断分离场景下保证模型有效性的方法流程示意图。
该实施例三是在上述实施例一的基础上，考虑针对同一个Analytics ID，需要多个子模型共同协作完成分析的场景。该场景下任意一个子模型的性能下降都会导致Analytics ID对应的模型性能下降，如果仅根据Analytics ID进行模型监控，则无法精确定位到子模型的性能，会将Analytics ID对应的所有子模型都进行再训练和更新，而实际可能某些子模型性能良好并不需要更新，这就导致了不必要的训练和更新。
本实施例考虑进一步地增加模型标识(model ID)来表示各个子模型。
该实施例包括以下步骤：
步骤701，训练NWDAF向NRF注册。
训练NWDAF向NRF发送NF注册请求，携带NF配置文件，该NF配置文件包括NF类型，NF服务名称(NF Service)，分析类型标识(Analytics ID)等信息，以及还包括模型标识(model ID)和模型索引信息。该模型索引信息可以是模型版本号(version)、位置信息(location)或URL等。其中，version表示模型版本，location或URL表示模型的存储位置，三者中使用任意一个都可以。可选的，当模型索引信息是location或URL，该location或URL中也可以包含version。该模型标识用于唯一标识一个模型，例如，该模型标识可以由NWDAF地址、PLMN ID、在某个NWDAF范围内唯一的modelID组成。
需要说明的是，NF配置文件可以携带多个模型索引信息，每个模型标识对应一个模型索引信息。
相应地，NRF存储NF配置文件以及向训练NWDAF发送NF注册响应。
可选的，该更新的NF配置文件还可以携带多个模型标识，每个模型标识用于标识多个更新的模型中的一个模型。
步骤702，推断NWDAF向NRF注册。
推断NWDAF向NRF发送NF注册请求，携带NF配置文件，该NF配置文件包括NF类型，NF服务名称(NF Service)，分析类型标识(Analytics ID)等信息，以及还包括模型状态信息和模型标识。其中，每个模型标识对应一个模型状态信息。
模型状态信息用于指示模型标识对应的模型的使用状态。该模型状态信息的可选取值包括但不限于：
1)‘null’：表示没有模型可以使用。
2)‘ok’：表示模型性能良好，可以对外提供分析服务。
3)‘limited’：表示模型性能有所下降，但仍可以提供服务，需要进行再训练。
4)‘stopped’：表示模型关闭，已停止提供服务。
其中，在注册过程中，NF注册请求携带的模型状态信息为‘null’，即推断NWDAF上当前没有模型可以使用。
示例性地，NF配置文件携带的模型状态信息和模型标识如下：
{(model ID 1，null)，(model ID 2，null)，(model ID 3，null)}。
或者，NF配置文件携带的模型状态信息和模型标识如下：
{(model ID 1，model ID 2，model ID 3)，(null，null，null)}。
可选的，在实际应用中NF配置文件携带的模型状态信息和模型标识可以是一个列表，该列表包含多个项信息，每个项信息包含一个模型状态信息和一个模型标识。
相应地，NRF存储NF配置文件以及向推断NWDAF发送NF注册响应。
步骤703，推断NWDAF发现训练NWDAF，并向训练NWDAF请求获取模型信息。
该过程可以参考图5实施例中的步骤503，不再赘述。基于该过程，推断NWDAF可以从训练NWDAF中获取到模型信息。
步骤704，推断NWDAF向NRF订阅训练NWDAF的状态。
后续训练NWDAF在NRF中注册的NF配置文件发生更新时，NRF通知推断NWDAF。
步骤705，训练NWDAF向NRF订阅推断NWDAF的状态。
后续推断NWDAF在NRF中注册的NF配置文件发生更新时，NRF通知训练NWDAF。
上述步骤704与步骤705相互之间没有固定的先后顺序。
步骤706，推断NWDAF向NRF发送NF更新请求，携带更新的NF配置文件。
该更新的NF配置文件至少携带更新的模型状态信息，该更新的模型状态信息比如可以是‘ok’。其中，每个模型标识对应一个更新的模型状态信息。
可选的，该更新的NF配置文件还携带Analytics ID，用于标识要更新的模型。可选的，该更新的NF配置文件还携带NF类型，NF服务名称(NF Service)等。
可选的，该更新的NF配置文件还可以携带模型标识，用于标识更新的模型。
步骤707，NRF更新存储的NF配置文件。
步骤708，NRF向推断NWDAF发送NF更新响应。
该NF更新响应用于通知NF配置文件更新成功。
步骤709，NRF向训练NWDAF发送NF状态更新通知，其中携带更新的模型状态信息。
其中，每个模型标识对应一个更新的模型状态信息。更新的模型状态信息比如可以是‘ok’。
可选的，该NF状态更新通知还携带指示信息，用于指示更新的类型是模型状态信息更新。
可选的，该NF状态更新通知还可以携带模型标识，用于标识更新的模型。
由于上述步骤705中，训练NWDAF向NRF订阅了推断NWDAF的状态，因此当NRF中存储的推断NWDAF的NF配置文件发生更新后，NRF通知训练NWDAF。
步骤710，推断NWDAF确定模型需要再训练。
判断依据可以是对模型性能的评估结果(比如模型精度下降)，也可以是NF上报的业务KPI(比如KPI下降)。
需要说明的是，该步骤710发生在推断NWDAF中的模型运行过程中，出现的时间不固定。
需要说明的是，该步骤中，判断的结果可以是：某一个或某几个子模型需要再训练。比如针对某个Analytics ID对应的模型，一共有10个子模型，分别用model ID 1至model ID10表示。该步骤710的判断结果比如是：model ID 1至model ID 3对应的子模型需要再训练，model ID 4至model ID 10对应的子模型不需要再训练。
步骤711，推断NWDAF向NRF发送NF更新请求，携带更新的NF配置文件。
该更新的NF配置文件至少携带更新的模型状态信息，其中，每个模型标识对应一个更新的模型状态信息。该更新的模型状态信息比如可以是‘limited’。可选的，该更新的NF配置文件还携带Analytics ID，用于标识要更新的模型。可选的，该更新的NF配置文件还携带NF类型，NF服务名称(NF Service)等。
可选的，该更新的NF配置文件还可以携带模型标识，用于标识更新的模型。
NRF更新存储的NF配置文件，然后向推断NWDAF发送NF更新响应。
需要说明的是，该步骤711中的更新的NF配置文件中携带的模型标识即为上述步骤710中确定的需要再训练的子模型的标识信息，更新的模型状态信息即为需要再训练的子模型的标识信息对应的更新的模型状态信息。
步骤712，NRF向训练NWDAF发送NF状态更新通知，其中携带更新的模型状态信息。
其中，每个模型标识对应一个更新的模型状态信息。该更新的模型状态信息可以是‘limited’。
可选的，该NF状态更新通知还携带指示信息，用于指示更新的类型是模型状态信息更新。
可选的，该NF状态更新通知还可以携带模型标识，用于标识更新的模型。
由于上述步骤705中，训练NWDAF向NRF订阅了推断NWDAF的状态，因此当NRF中存储的推断NWDAF的NF配置文件发生更新后，NRF通知训练NWDAF。
步骤713，训练NWDAF对模型启动再训练。
训练NWDAF对模型启动再训练，得到训练后的模型以及相应的模型索引信息，如模型版本号、位置信息或URL等。
需要说明的是，该步骤中，仅对接收到的需要进行训练的子模型进行启动再训练。比如接收到的模型标识是model ID 1至model ID 3，则对model ID 1至model ID 3对应的子 模型进行再训练。
步骤714，训练NWDAF向NRF发送NF更新请求，携带更新的NF配置文件。
该更新的NF配置文件至少携带更新的模型索引信息，每个模型标识对应一个更新的模型状态信息。该更新的模型索引信息比如可以是更新的模型版本信息、更新的模型位置信息或更新的模型URL等。可选的，该更新的NF配置文件还携带Analytics ID，用于标识要更新的模型。可选的，该更新的NF配置文件还携带NF类型，NF服务名称(NF Service)等。
可选的，该更新的NF配置文件还可以携带模型标识，用于标识更新的模型。
NRF更新存储的NF配置文件，然后向训练NWDAF发送NF更新响应。
步骤715，NRF向推断NWDAF发送NF状态更新通知，其中携带更新的模型索引信息。
该更新的模型索引信息比如可以是更新的模型版本信息、更新的模型位置信息或更新的模型URL等。
可选的，该NF状态更新通知还携带指示信息，用于指示更新的类型是模型索引信息更新。
可选的，该NF状态更新通知还携带模型标识，用于标识更新的模型。
由于上述步骤704中，推断NWDAF向NRF订阅了训练NWDAF的状态，因此当NRF中存储的训练NWDAF的NF配置文件发生更新后，NRF通知推断NWDAF。
步骤716，推断NWDAF向训练NWDAF发送模型请求，携带Analytics ID和更新的模型索引信息。
Analytics ID用于指示与该Analytics ID对应的模型。
模型标识用于指示该Analytics ID对应的模型中的子模型。
步骤717，训练NWDAF向推断NWDAF发送模型响应，携带模型信息。
该模型信息包括与更新后的模型标识对应的模型信息，也即获取到新的模型对应的模型信息。
其中，模型信息的具体实现，可以参考前述实施例的描述。
步骤718，推断NWDAF进行模型更新。
也即，推断NWDAF根据接收到的新的模型信息，对正在使用的旧模型(具体是相应需要更新的子模型)进行更新。
可选的，推断NWDAF在模型更新前，对新的模型信息进行本地测试，测试通过后再进行更新或者替换。
步骤719，推断NWDAF向NRF发送NF更新请求，携带更新的NF配置文件。
该更新的NF配置文件至少携带更新的模型状态信息，每个模型标识对应一个更新的模型状态信息。该更新的模型状态信息比如可以是‘ok’。可选的，该更新的NF配置文件还携带Analytics ID，用于标识要更新的模型。可选的，该更新的NF配置文件还携带NF类型，NF服务名称(NF Service)等。
NRF更新存储的NF配置文件，然后向推断NWDAF发送NF更新响应。
可选的，该更新的NF配置文件还可以携带模型标识，用于标识更新的模型。
步骤720，NRF向训练NWDAF发送NF状态更新通知，其中携带更新的模型状态信息。
其中，每个模型标识对应一个更新的模型状态信息。更新的模型状态信息可以是‘ok’。
可选的，该NF状态更新通知还携带指示信息，用于指示更新的类型是模型状态信息更新。
可选的，该NF状态更新通知还可以携带模型标识，用于标识更新的模型。
其中，上述步骤714-步骤720可选步骤。例如，如果步骤713中训练NWDAF确定模型不需要再训练或者训练NWDAF可以容忍模型性能下降到模型性能要求以下或者训练NWDAF上当前不具备模型再训练的能力(如硬件资源有限)，步骤714-步骤720可以不执行。
由于上述步骤705中，训练NWDAF向NRF订阅了推断NWDAF的状态，因此当NRF中存储的推断NWDAF的NF配置文件发生更新后，NRF通知训练NWDAF。
基于上述实施例，增加了模型标识(也称为子模型标识)，按照子模型粒度进行性能监控，在一个Analytics ID对应多个子模型的场景下可以实现精准的执行模型再训练与更新，避免浪费训练和传输资源。
实施例四
如图8所示，为本申请实施例提供的又一种训练-推断分离场景下保证模型有效性的方法流程示意图。
上述实施例一至实施例三是考虑通过NRF实现训练NWDAF和推断NWDAF的信息交互，该实施例四考虑在训练NWDAF和推断NWDAF之间的接口上新增操作，直接交互信息。
该实施例包括以下步骤：
步骤801，训练NWDAF向NRF注册。
训练NWDAF向NRF发送NF注册请求，携带NF配置文件，该NF配置文件包括NF类型，NF服务名称(NF Service)，分析类型标识(Analytics ID)等信息。
相应地，NRF存储NF配置文件以及向训练NWDAF发送NF注册响应。
步骤802a，推断NWDAF向NRF发送NF发现请求，携带NF配置文件。
比如携带的NF配置文件包含NF类型(如NWDAF)，NF服务名称(如ModelProvision)，以及携带Analytics ID，则该NF发现请求用于请求从NRF获取对应该Analytics ID的训练NWDAF。
步骤802b，NRF向推断NWDAF发送NF发现响应，其中携带NWDAF实例。
其中，携带的NWDAF实例是训练NWDAF的一个实例，可以用训练NWDAF的ID或IP地址来表示。
步骤803a，推断NWDAF向训练NWDAF发送模型请求，其中携带Analytics ID。
其中，推断NWDAF可以基于从NRF获取到的训练NWDAF的ID或IP地址，向训练NWDAF发送模型请求，携带的Analytics ID用于指示请求获取与该Analytics ID对应的模型。
步骤803b，训练NWDAF向推断NWDAF发送模型响应，携带模型信息。
其中，模型信息的具体实现，可以参考前述实施例的描述。
例如，上述步骤801-步骤803b可选步骤。例如，如果推断NWDAF中配置了训练NWDAF的NF配置文件，则步骤801-步骤803b可以不执行。
步骤804a，训练NWDAF向推断NWDAF发送模型性能信息订阅请求，其中携带Analytics ID，模型性能指标(如精确率(Precision)、准确率(Accuracy)、错误率(Error Rate)、召回率(Recall)、F1分数(F-Score)、均方误差(Mean Square Error,MSE)、均方根误差(Root Mean Squared Error,RMSE)、均方根对数误差(Root Mean Squared Logarithmic Error,RMSLE)、平均绝对误差(Mean Absolute Error,MAE)、模型推理时长、模型鲁棒性、模型可扩展性、模型可解释性)以及上报周期。
其中，精确率、准确率、错误率、召回率、F1分数用于指示分类类型或者标注类型的模型的性能。均方误差、均方根误差、均方根对数误差、平均绝对误差用于指示回归类模型的性能。模型推理时长用于指示模型预测需要的时间。模型鲁棒性用于指示模型处理缺失值和异常值的能力。模型可拓展性用于指示处理大数据集的能力。模型可解释性用于指示模型预测标准的可理解性，比如，决策树模型由于产生的规则或者树结构导致模型可解释性高，神经网络模型由于存在大量模型参数导致模型可解释性低。
步骤804b，推断NWDAF向训练NWDAF发送模型性能信息通知，其中携带Analytics ID，模型性能指标以及模型性能指标对应的取值。
其中，推断NWDAF是基于上报周期，周期性地向训练NWDAF发送模型性能信息通知。
基于上述步骤804a至步骤804b，推断NWDAF可以周期性地向训练NWDAF上报模型性能信息。
可选的，模型性能信息通知中还可以携带推断NWDAF对模型的模型性能要求，和/或，推断NWDAF进行模型评估使用的数据等。其中，模型性能要求可以辅助训练NWDAF判断是否需要进行再训练以及判断再训练得到的模型性能是否满足推断NWDAF要求，推断NWDAF进行模型评估使用的数据包括模型的输入数据、模型的输出数据(推断结果)、与推断结果对应的网络实际测量值(网络数据)，可供训练NWDAF对模型进行再训练时使用。
可选的，上述步骤804a至步骤804b也可以由以下步骤804a’至步骤804b’替换。
步骤804a’，训练NWDAF向推断NWDAF发送模型性能信息订阅请求，其中携带Analytics ID，模型性能指标(如精确率、准确率、错误率、召回率、F1分数、均方误差、均方根误差、均方根对数误差、平均绝对误差、模型推理时长、模型鲁棒性、模型可扩展性、模型可解释性)以及性能门限值。
步骤804b’，推断NWDAF向训练NWDAF发送模型性能再训练通知，其中携带Analytics ID。
基于上述步骤804a’至步骤804b’，推断NWDAF确定模型性能指标对应的取值达到性能门限值，则向训练NWDAF上报模型性能信息通知，该模型性能再训练通知用于触发训练NWDAF对模型再训练。可选的，上述步骤804b’可以不携带性能门限值，则可以由推断NWDAF自行确定性能门限值。该步骤804b’的模型性能再训练通知也可以称为模型性能达到门限值通知或模型性能信息通知。
可选的，模型性能信息通知中还可以携带推断NWDAF对模型的模型性能要求，和/或，推断NWDAF进行模型评估使用的数据等。其中，模型性能要求可以是推断NWDAF自行确定的门限值，用于辅助训练NWDAF判断是否需要进行再训练以及判断再训练得到的模型性能是否满足推断NWDAF要求，推断NWDAF进行模型评估使用的数据包括模型 的输入数据、模型的输出数据(推断结果)、与推断结果对应的网络实际测量值(网络数据)，可供训练NWDAF对模型进行再训练时使用。
可选的，上述步骤804a至步骤804b也可以由以下步骤804a”至步骤804b”替换。
步骤804a”，训练NWDAF向推断NWDAF发送模型性能信息请求，其中携带Analytics ID，模型性能指标(如精确率、准确率、错误率、召回率、F1分数、均方误差、均方根误差、均方根对数误差、平均绝对误差、模型推理时长、模型鲁棒性、模型可扩展性、模型可解释性)。
步骤804b”，推断NWDAF向训练NWDAF发送模型性能信息响应，其中携带Analytics ID，模型性能指标以及模型性能指标对应的取值。
基于上述步骤804a”至步骤804b”，训练NWDAF可以周期性地向推断NWDAF发送模型性能信息请求，推断NWDAF每次收到模型性能信息请求，则基于模型性能指标进行模型性能评估，并向训练NWDAF发送模型性能信息响应。
可选的，模型性能信息响应中还可以携带推断NWDAF对模型的模型性能要求，和/或，推断NWDAF进行模型评估使用的数据等。其中，模型性能要求可以辅助训练NWDAF判断是否需要进行再训练以及判断再训练得到的模型性能是否满足推断NWDAF要求，推断NWDAF进行模型评估使用的数据包括模型的输入数据、模型的输出数据(推断结果)、与推断结果对应的网络实际测量值(网络数据)，可供训练NWDAF对模型进行再训练时使用。
可选的，上述步骤804a至步骤804b也可以由以下步骤804a”’至步骤804b”’替换。
步骤804a”’，训练NWDAF向推断NWDAF发送模型性能数据订阅请求，其中携带Analytics ID，以及上报周期。
步骤804b”’，推断NWDAF向训练NWDAF发送模型性能数据通知，其中携带Analytics ID和模型性能评估参考信息。
其中，模型性能评估参考信息包括模型的输入数据、模型的输出数据(推断结果)或与推断结果对应的网络实际测量值中的至少一项。
基于上述步骤804a”’至步骤804b”’，推断NWDAF基于上报周期，周期性地向训练NWDAF发送模型性能数据通知，即推断NWDAF可以周期性地向训练NWDAF上报模型性能评估参考信息。
其中，与推断结果对应的网络实际测量值(网络数据)可以由推断NWDAF从现网中采集后上报给训练NWDAF，也可以由训练NWDAF自行从现网中采集。
可选的，模型性能数据通知中还可以携带推断NWDAF对模型的模型性能要求。
训练NWDAF可以根据推断NWDAF周期性上报的模型性能评估参考信息，构建测试集并进行模型性能评估。
可选的，上述步骤804a至步骤804b也可以由以下步骤804a””至步骤804b””替换。
步骤804a””，训练NWDAF向推断NWDAF发送模型性能数据请求，其中携带Analytics ID。
可选的，模型性能数据请求中还包含时间范围，用于指示请求该时间范围内的性能数据。
步骤804b””，推断NWDAF向训练NWDAF发送模型性能数据响应，其中携带 Analytics ID和模型性能评估参考信息。
其中,模型性能评估参考信息包括模型的输入数据、模型的输出数据(推断结果)、或与推断结果对应的网络实际测量(网络数据)值中的至少一项。
基于上述步骤804a””至步骤804b””，训练NWDAF可以向推断NWDAF发送模型性能数据请求，推断NWDAF向训练NWDAF发送模型性能数据响应，即推断NWDAF向训练NWDAF发送模型性能评估参考信息，该模型性能评估参考信息可以是一定时间范围内的。
其中，与推断结果对应的网络实际测量值可以由推断NWDAF从现网中采集后上报给训练NWDAF，也可以由训练NWDAF自行从现网中采集。
可选的，模型性能数据响应中还可以携带推断NWDAF对模型的模型性能要求。
训练NWDAF可以根据推断NWDAF发送的模型性能评估参考信息构建测试集并进行模型性能评估。
步骤805，训练NWDAF确定启动模型再训练。
比如，若执行上述步骤804a至步骤804b，则训练NWDAF确定模型性能指标对应的取值达到训练NWDAF预设的性能门限值或者不满足推断NWDAF的模型性能要求，则确定启动模型再训练。
再比如，若执行上述步骤804a’至步骤804b’，则训练NWDAF收到模型性能信息通知，则确定启动模型再训练。
比如，若执行上述步骤804a”至步骤804b”，则训练NWDAF确定模型性能指标对应的取值达到训练NWDAF预设的性能门限值或者不满足推断NWDAF的模型性能要求，则确定启动模型再训练。
再比如，若执行上述步骤804a”’至步骤804b”’，或执行上述步骤804a””至步骤804b””，则训练NWDAF根据模型性能评估参考信息，确定模型性能达到训练NWDAF预设的性能门限值，或者不满足推断NWDAF的模型性能要求，则确定启动模型再训练。
步骤806，训练NWDAF向推断NWDAF发送模型更新请求，携带Analytics ID以及新的模型信息。
可选的，模型更新请求中的新模型信息可以是新模型的参数项取值、或新模型文件或包含新模型的镜像文件、或新模型的地址(如URL或IP地址)。
需要说明的是，如果步骤806中携带的是新模型的地址，则推断NWDAF可以根据该地址获取包含新模型信息的文件，该文件可以是包含新模型的参数项取值的文件、或是模型文件、或是包含新模型的镜像文件。
步骤807，推断NWDAF向训练NWDAF发送模型更新响应。
步骤808，推断NWDAF进行模型更新。
也即，推断NWDAF根据接收到的新的模型信息，对正在使用的旧模型进行更新或者替换。
可选的，推断NWDAF在模型更新前，对新的模型信息进行本地测试，测试通过后再进行更新或者替换。
其中，上述步骤806-步骤808可选步骤。例如，如果步骤805中训练NWDAF确定模型不需要再训练或者训练NWDAF可以容忍模型性能下降到模型性能要求以下或者训练NWDAF上当前不具备模型再训练的能力(如硬件资源有限)，步骤806-步骤808可以不 执行。
基于上述实施例，训练NWDAF向推断NWDAF发送模型性能订阅或模型性能请求，实现对推断NWDAF中模型性能的监控，当性能下降满足再训练条件时，训练NWDAF及时进行再训练，并将新模型发送给推断NWDAF用于更新，保证了推断NWDAF中模型的模型性能。
实施例五
如图9所示，为本申请实施例提供的又一种训练-推断分离场景下保证模型有效性的方法流程示意图。
该实施例五是在实施例四的基础上，考虑有多个推断NWDAF存在的场景，具体场景同实施例二，可参考实施例二的场景描述。类似实施例二，本实施例考虑在模型更新请求中增加指示模型性能信息的参数，包括精度、所需计算量等，帮助其它暂不需要更新的推断NWDAF判断是否需要请求新模型。
该实施例包括以下步骤：
步骤901至步骤905，类似于上述实施例四中的步骤801至步骤805。
需要说明的是，该步骤901至步骤905中，涉及推断NWDAF1的相关操作以及推断NWDAF2的相关操作，均可以分别参考上述步骤802a至步骤804b中关于推断NWDAF的相关操作。并且，步骤905中，训练NWDAF是根据推断NWDAF1发送的模型性能信息通知或模型性能信息响应，触发启动模型再训练。
接下来，训练NWDAF需要通知推断NWDAF执行模型更新。
第一种方案是：不区分推断NWDAF，即训练NWDAF始终将训练得到的新的模型信息发送给所有推断NWDAF。该方案参考以下步骤906a至步骤906b。
第二种方案是：区分不同的推断NWDAF，仅向触发训练NWDAF执行模型训练的推断NWDAF发送新的模型信息。该方案参考以下步骤907a至步骤907c。
需要说明的是，上述第一种方案与第二种方案，二者选择一种进行执行。
第一种方案：
步骤906a，训练NWDAF向推断NWDAF1发送模型更新请求，携带Analytics ID，新的模型信息以及模型性能信息。
步骤906b，推断NWDAF1判断是否需要更新模型。
推断NWDAF1收到模型更新请求后，可以基于模型性能信息，和/或，对新模型信息的本地测试结果判断是否需要更新模型。如果确定需要更新，则使用新的模型信息更新或者替换旧模型。
步骤906c，训练NWDAF向推断NWDAF2发送模型更新请求，携带Analytics ID，新的模型信息以及模型性能信息。
步骤906d，推断NWDAF2判断是否需要更新模型。
推断NWDAF2收到模型更新请求后，可以基于模型性能信息，和/或，对新模型的本地测试结果判断是否需要更新模型。如果确定需要更新，则使用新的模型信息更新或者替换旧模型。
第二种方案：
步骤907a，训练NWDAF向推断NWDAF1发送模型更新请求，携带Analytics ID以及新的模型信息。
步骤907b，推断NWDAF1更新模型。
推断NWDAF1收到模型更新请求后，使用新的模型信息更新或者替换旧模型。
可选的，推断NWDAF在模型更新前，对新的模型信息进行本地测试，测试通过后再进行更新或者替换。
步骤907c，训练NWDAF向推断NWDAF2发送模型训练完成通知，携带Analytics ID以及模型性能信息。
步骤907d，推断NWDAF2判断是否需要更新模型。
比如，推断NWDAF2可以根据自身算力、模型性能要求，以及接收到的模型性能信息，决定是否需要更新模型。
当推断NWDAF2确定需要更新模型，则执行以下步骤907e至步骤907g，否则不执行以下步骤907e至步骤907g。
步骤907e，可选的，推断NWDAF2向训练NWDAF发送模型请求，携带Analytics ID。
Analytics ID用于指示与该Analytics ID对应的模型。
步骤907f，可选的(取决于步骤907e是否执行)，训练NWDAF向推断NWDAF2发送模型响应，携带新的模型信息。
步骤907g，推断NWDAF2更新模型。
也即，推断NWDAF2根据接收到的新的模型信息，对正在使用的旧模型进行更新。
可选的，推断NWDAF2在模型更新前，对新的模型信息进行本地测试，测试通过后再进行更新或者替换。
基于上述实施例，使用相同模型的推断NWDAF可以获取新模型的信息，并根据模型性能信息确定是否需要请求新模型，避免不必要的模型传输和本地评估过程。
实施例六
如图10所示，为本申请实施例提供的又一种训练-推断分离场景下保证模型有效性的方法流程示意图。
该实施例是在实施例四的基础上，考虑针对同一个Analytics ID，需要多个子模型共同协作完成分析的场景。类似实施例三的解决方案，本实施例考虑进一步地增加model ID来标识各个子模型，训练NWDAF给子模型分配不同的model ID，通过model ID精确监控每个子模型的性能。
该实施例包括以下步骤：
步骤1001，训练NWDAF向NRF注册。
训练NWDAF向NRF发送NF注册请求，携带NF配置文件，该NF配置文件包括NF类型，NF服务名称(NF Service)，分析类型标识(Analytics ID)等信息。
相应地，NRF存储NF配置文件以及向训练NWDAF发送NF注册响应。
步骤1002a，推断NWDAF向NRF发送NF发现请求，携带NF配置文件。
比如携带的NF配置文件包含NF类型(如NWDAF)，NF服务名称(如ModelProvision)，以及携带Analytics ID，则该NF发现请求用于请求从NRF获取对应该Analytics ID的训练 NWDAF。
步骤1002b，NRF向推断NWDAF发送NF发现响应，其中携带NWDAF实例。
其中，携带的NWDAF实例是训练NWDAF的一个实例，可以用训练NWDAF的ID或IP地址来表示。
步骤1003a，推断NWDAF向训练NWDAF发送模型请求，其中携带Analytics ID。
其中，推断NWDAF可以基于从NRF获取到的训练NWDAF的ID或IP地址，向训练NWDAF发送模型请求，携带的Analytics ID用于指示请求获取与该Analytics ID对应的模型。
步骤1003b，训练NWDAF向推断NWDAF发送模型响应，携带模型信息和模型标识。
其中，模型信息的具体实现，可以参考前述实施例的描述。
其中，每个模型标识对应一个模型信息。
可选的，模型信息和模型标识可以以模型列表形式实现，也即模型响应携带模型列表，模型列表包含模型信息与模型标识以及模型信息与模型标识之间的对应关系。示例性地，模型列表包括：<模型信息1，模型标识1>，<模型信息2，模型标识2>，……。
其中，上述步骤1001-步骤1003b可选步骤行。例如，如果推断NWDAF中配置了训练NWDAF的NF配置文件，则步骤1001-步骤1003b可以不执行。
步骤1004a，训练NWDAF向推断NWDAF发送模型性能信息订阅请求，其中携带Analytics ID，模型性能指标(如精确率、准确率、错误率、召回率、F1分数、均方误差、均方根误差、均方根对数误差、平均绝对误差、模型推理时长、模型鲁棒性、模型可扩展性、模型可解释性)、上报周期以及模型标识。
需要说明的是，模型性能信息订阅请求可以携带多个模型标识以及每个模型标识对应的模型性能指标、上报周期。特别地，当每个模型标识对应的上报周期相同时，则可以只携带一个上报周期。
步骤1004b，推断NWDAF向训练NWDAF发送模型性能信息通知，其中携带Analytics ID，模型性能指标以及模型性能指标对应的取值。
其中，推断NWDAF是基于上报周期，周期性地向训练NWDAF发送各个子模型对应的模型性能信息通知。
基于上述步骤1004a至步骤1004b，推断NWDAF可以周期性地向训练NWDAF上报各个子模型对应的模型性能信息。
可选的，模型性能信息通知中还可以携带推断NWDAF对各子模型的模型性能要求，和/或，推断NWDAF进行各子模型评估使用的数据等。其中，模型性能要求可以辅助训练NWDAF判断是否需要进行再训练以及判断再训练得到的模型性能是否满足推断NWDAF要求，推断NWDAF进行模型评估使用的数据包括模型的输入数据、模型的输出数据(推断结果)、与推断结果对应的网络实际测量值，可供训练NWDAF对模型进行再训练时使用。
可选的，上述步骤1004a至步骤1004b也可以由以下步骤1004a’至步骤1004b’替换。
步骤1004a’，训练NWDAF向推断NWDAF发送模型性能信息订阅请求，其中携带Analytics ID，模型性能指标(如精确率、准确率、错误率、召回率、F1分数、均方误差、均方根误差、均方根对数误差、平均绝对误差、模型推理时长、模型鲁棒性、模型可扩展性、模型可解释性)、性能门限值以及模型标识。
需要说明的是，模型性能信息订阅请求可以携带多个模型标识以及每个模型标识对应的模型性能指标、性能门限值。特别地，当每个模型标识对应的性能门限值相同时，则可以只携带一个性能门限值。
步骤1004b’，推断NWDAF向训练NWDAF发送模型性能再训练通知，其中携带Analytics ID。
基于上述步骤1004a’至步骤1004b’，推断NWDAF确定子模型的模型性能指标对应的取值达到性能门限值，则向训练NWDAF上报子模型对应的模型性能信息通知，该模型性能再训练通知用于触发训练NWDAF对子模型再训练。可选的，上述步骤1004b’可以不携带性能门限值，则可以由推断NWDAF自行确定性能门限值。该步骤1004b’的模型性能再训练通知也可以称为模型性能达到门限值通知或模型性能信息通知。
可选的，模型性能信息通知中还可以携带推断NWDAF对子模型的模型性能要求，和/或，推断NWDAF进行模型评估使用的数据等。其中，模型性能要求可以是推断NWDAF自行确定的门限值，用于辅助训练NWDAF判断是否需要进行再训练以及判断再训练得到的模型性能是否满足推断NWDAF要求，推断NWDAF进行模型评估使用的数据包括模型的输入数据、模型的输出数据(推断结果)、与推断结果对应的网络实际测量值，可供训练NWDA对模型进行再训练时使用。
可选的，上述步骤1004a至步骤1004b也可以由以下步骤1004a”至步骤1004b”替换。
步骤1004a”，训练NWDAF向推断NWDAF发送模型性能信息请求，其中携带Analytics ID，模型性能指标(如精确率、准确率、错误率、召回率、F1分数、均方误差、均方根误差、均方根对数误差、平均绝对误差、模型推理时长、模型鲁棒性、模型可扩展性、模型可解释性)以及模型标识。
需要说明的是，模型性能信息请求可以携带多个模型标识以及每个模型标识对应的模型性能指标。
步骤1004b”，推断NWDAF向训练NWDAF发送模型性能信息响应，其中携带Analytics ID，模型性能指标以及模型性能指标对应的取值。
基于上述步骤1004a”至步骤1004b”，训练NWDAF可以周期性地向推断NWDAF发送模型性能信息请求，推断NWDAF每次收到模型性能信息请求，则基于模型性能指标进行模型性能评估，并向训练NWDAF发送子模型对应的模型性能信息响应。
可选的，模型性能信息响应中还可以携带推断NWDAF对各子模型的模型性能要求，和/或，推断NWDAF进行模型评估使用的数据等。其中，模型性能要求可以辅助训练NWDAF判断是否需要进行再训练以及判断再训练得到的模型性能是否满足推断NWDAF要求，推断NWDAF进行模型评估使用的数据包括模型的输入数据、模型的输出数据(推断结果)、与推断结果对应的网络实际测量值(网络数据)，可供训练NWDAF对模型进行再训练时使用。
可选的，上述步骤1004a至步骤1004b也可以由以下步骤1004a”’至步骤1004b”’替换。
步骤1004a”’，训练NWDAF向推断NWDAF发送模型性能数据订阅请求，其中携带Analytics ID，上报周期以及模型标识。
需要说明的是，模型性能数据订阅请求可以携带多个模型标识以及每个模型标识对应 的上报周期。
步骤1004b”’，推断NWDAF向训练NWDAF发送模型性能数据通知，其中携带Analytics ID和模型性能评估参考信息。
其中，模型性能评估参考信息包括模型的输入数据、模型的输出数据(推断结果)或与推断结果对应的网络实际测量值(网络数据)中的至少一项。
需要说明的是，这里的模型性能评估参考信息可以是多个模型性能评估参考信息，具体的，每个模型标识对应一个模型性能评估参考信息。
基于上述步骤1004a”’至步骤1004b”’，推断NWDAF基于上报周期，周期性地向训练NWDAF发送模型性能数据通知，即推断NWDAF可以周期性地向训练NWDAF上报各个子模型分别对应的模型性能评估参考信息。
其中，与推断结果对应的网络实际测量值(网络数据)可以由推断NWDAF从现网中采集后上报给训练NWDAF，也可以由训练NWDAF自行从现网中采集。
可选的，模型性能数据通知中还可以携带推断NWDAF对各个子模型的模型性能要求。
训练NWDAF可以根据推断NWDAF周期性上报的模型性能评估参考信息，构建测试集并进行模型性能评估。
可选的，上述步骤1004a至步骤1004b也可以由以下步骤1004a””至步骤1004b””替换。
步骤1004a””，训练NWDAF向推断NWDAF发送模型性能数据请求，其中携带Analytics ID以及模型标识。
需要说明的是，模型性能数据订阅请求可以携带多个模型标识。
可选的，模型性能数据请求中还包含时间范围，用于指示请求该时间范围内的性能数据。具体的，每个模型标识可以对应一个时间范围。
步骤1004b””，推断NWDAF向训练NWDAF发送模型性能数据响应，其中携带Analytics ID和模型性能评估参考信息。
其中,，模型性能评估参考信息包括模型的输入数据、模型的输出数据(推断结果)，和/或，或与推断结果对应的网络实际测量值(网络数据)中的至少一项。
需要说明的是，这里的模型性能评估参考信息可以是多个模型性能评估参考信息，具体的，每个模型标识对应一个模型性能评估参考信息。
基于上述步骤1004a””至步骤1004b””，训练NWDAF可以向推断NWDAF发送模型性能数据请求，推断NWDAF向训练NWDAF发送模型性能数据响应，即推断NWDAF向训练NWDAF发送模型性能评估参考信息，该模型性能评估参考信息可以是一定时间范围内的。
其中，与推断结果对应的网络实际测量值(网络数据)可以由推断NWDAF从现网中采集后上报给训练NWDAF，也可以由训练NWDAF自行从现网中采集。
可选的，模型性能数据响应中还可以携带推断NWDAF对模型的模型性能要求。
训练NWDAF可以根据推断NWDAF发送的模型性能评估参考信息构建测试集并进行模型性能评估。
步骤1005，训练NWDAF确定启动模型再训练。
比如，若执行上述步骤1004a至步骤1004b，则训练NWDAF确定模型性能指标对应的取值达到训练NWDAF预设的性能门限值或者不满足推断NWDAF的模型性能要求，则 确定启动模型再训练。
再比如，若执行上述步骤1004a’至步骤1004b’，则训练NWDAF收到模型性能信息通知，则确定启动模型再训练。
比如，若执行上述步骤1004a”至步骤1004b”，则训练NWDAF确定模型性能指标对应的取值达到训练NWDAF预设的性能门限值或者不满足推断NWDAF的模型性能要求，则确定启动模型再训练。
再比如，若执行上述步骤1004a”’至步骤1004b”’，或执行上述步骤1004a””至步骤1004b””，则训练NWDAF根据模型性能评估参考信息，确定模型性能达到训练NWDAF预设的性能门限值，或者不满足推断NWDAF的模型性能要求，则确定启动模型再训练。
步骤1006，训练NWDAF向推断NWDAF发送模型更新请求，携带Analytics ID、新的模型信息以及模型标识。
需要说明的是，模型更新请求可以携带多个模型标识以及每个模型标识对应的新的模型信息。
步骤1007，推断NWDAF向训练NWDAF发送模型更新响应。
步骤1008，推断NWDAF进行模型更新。
也即，推断NWDAF根据接收到的新的模型信息，对正在使用的旧模型(具体是旧的子模型)进行更新或者替换。
可选的，推断NWDAF在模型更新前，对新的模型信息进行本地测试，测试通过后再进行更新或者替换。
其中，上述步骤1006-步骤1008可选步骤。例如，如果步骤805中训练NWDAF确定模型不需要再训练或者训练NWDAF可以容忍模型性能下降到模型性能要求以下或者训练NWDAF上当前不具备模型再训练的能力(如硬件资源有限)，步骤1006-步骤1008可以不执行。
基于上述实施例，通过增加子模型的标识，按照模型粒度进行性能监控，在一个Analytics ID对应多个子模型的场景下可以实现精准的模型再训练与更新，避免浪费训练和传输资源。
实施例七
如图11所示，为本申请实施例提供的又一种训练-推断分离场景下保证模型有效性的方法流程示意图。
该实施例考虑由训练NWDAF进行周期性再训练，并通知推断NWDAF有可用的新模型。该实施例适用于推断NWDAF不具备评估功能的场景，即无法获取推断NWDAF关于模型性能的实时反馈。为了维持模型的性能，训练NWDAF可以周期性的进行再训练。
该实施例包括以下步骤：
步骤1101，训练NWDAF向NRF注册。
训练NWDAF向NRF发送NF注册请求，携带NF配置文件，该NF配置文件包括NF类型，NF服务名称(NF Service)，分析类型标识(Analytics ID)等信息，以及还包括模型索引信息。该模型索引信息可以是模型版本号(version)、位置信息(location)或统一资源定位符(Uniform Resource Locator，URL)等。其中，version表示模型版本，location 或URL表示模型的存储位置，三者中使用任意一个都可以。可选的，当模型索引信息是location或URL，该location或URL中也可以包含version。
相应地，NRF存储NF配置文件以及向训练NWDAF发送NF注册响应。
步骤1102，推断NWDAF发现训练NWDAF，并向训练NWDAF请求获取模型信息。
该过程可以参考图3实施例中的步骤304至步骤307，不再赘述。基于该过程，推断NWDAF可以从训练NWDAF中获取到模型信息。
步骤1103，推断NWDAF向NRF订阅训练NWDAF的状态。
后续训练NWDAF在NRF中注册的NF配置文件发生更新时，NRF通知推断NWDAF。
步骤1104，训练NWDAF周期性地启动模型再训练。
比如，训练NWDAF可以设置定时器，每隔固定的时间再训练一次。
步骤1105，训练NWDAF向NRF发送NF更新请求，携带更新的NF配置文件。
该更新的NF配置文件至少携带更新的模型索引信息。可选的，该更新的NF配置文件还携带NF类型，NF服务名称(NF Service)等。
可选的，该更新的NF配置文件中还可以携带更新的模型性能信息，如模型准确度、达到该准确度所需的硬件能力信息、模型推理所需计算量、模型的推理时长、模型的大小等。
步骤1106，NRF更新存储的NF配置文件。
步骤1107，NRF向训练NWDAF发送NF更新响应。
该NF更新响应用于通知NF配置文件更新成功。
步骤1108，NRF向推断NWDAF发送NF状态更新通知，其中携带更新的模型索引信息。
可选的，该NF状态更新通知还携带指示信息，用于指示更新的类型是模型索引信息更新。
步骤1109，推断NWDAF向训练NWDAF发送模型请求，携带Analytics ID和更新的模型索引信息。
Analytics ID用于指示与该Analytics ID对应的模型。
步骤1110，训练NWDAF向推断NWDAF发送模型响应，携带模型信息。
该模型信息包括与更新的模型索引信息对应的模型信息，也即获取到新的模型对应的模型信息。
步骤1111，推断NWDAF进行模型更新。
也即，推断NWDAF根据接收到的新的模型信息，对正在使用的旧模型进行更新。
可选的，推断NWDAF在模型更新前，对新的模型信息进行本地测试，测试通过后再进行更新或者替换。
其中，上述步骤1109-步骤1111可选步骤。例如，如果步骤1108后中推理NWDAF可以容忍模型性能下降到模型性能要求，步骤1109-步骤1111可以不执行。
需要说明的是，上述步骤1104-步骤1108是周期性进行的，因此步骤1109-步骤1111是可选的，因为训练NWDAF只负责进行周期性再训练，是否请求新的模型进行更新由推断NWDAF自行确定。
基于该实施例，在推断NWDAF不具备评估功能的场景下，即训练NWDAF无法获取推断NWDAF关于模型性能的实时反馈时可以进行周期性再训练，以保证模型的性能。
需要说明的是，基于上述实施例七，在当有多个推断NWDAF时，上述步骤1107之后，NRF可以向多个推断NWDAF发送NF状态更新通知，从而使得多个推断NWDAF均可以向训练NWDAF发送模型请求，从而实现多个推断NWDAF的模型更新。
需要说明的是，基于上述实施例七，当一个分析类型标识对应多个子模型，每个子模型用一个模型标识进行标识，则上述步骤1105中还可以携带一个或多个模型标识，以及上述步骤1108中，还可以携带该一个或多个模型标识，进而上述步骤1109中可以携带该一个或多个模型标识，从而实现推断NWDAF中的一个或多个子模型的更新。
实施例八
如图12所示，为本申请实施例提供的又一种训练-推断分离场景下保证模型有效性的方法流程示意图。
该实施例八与实施例七场景相同，即推断NWDAF不具备评估功能时，由训练NWDAF进行周期性再训练，并给推断NWDAF发送模型更新消息。
该实施例包括以下步骤：
步骤1201，训练NWDAF向NRF注册。
训练NWDAF向NRF发送NF注册请求，携带NF配置文件，该NF配置文件包括NF类型，NF服务名称(NF Service)，分析类型标识(Analytics ID)等信息。
相应地，NRF存储NF配置文件以及向训练NWDAF发送NF注册响应。
步骤1202，推断NWDAF发现训练NWDAF，并向训练NWDAF请求获取模型。
该过程可以参考图3实施例中的步骤304至步骤307，不再赘述。基于该过程，推断NWDAF可以从训练NWDAF中获取到模型信息。
步骤1203，训练NWDAF周期性地启动模型再训练。
比如，训练NWDAF可以设置定时器，每隔固定的时间再训练一次。
步骤1204，训练NWDAF向推断NWDAF发送模型更新请求，携带Analytics ID以及新的模型信息。
可选的，模型更新请求中还可以携带新模型的性能信息，如模型准确度、达到该准确度所需的硬件能力信息、模型推理所需计算量、模型的推理时长、模型的大小等。
步骤1205，推断NWDAF向训练NWDAF发送模型更新响应。
步骤1206，推断NWDAF进行模型更新。
也即，推断NWDAF根据接收到的新的模型信息，对正在使用的旧模型进行更新。
可选的，推断NWDAF在模型更新前，对新的模型信息进行本地测试，测试通过后再进行更新或者替换。
其中，上述步骤1204-步骤1206可选步骤。例如，如果步骤1203中训练NWDAF确定训练更新后的模型的模型性能评估结果小于或者等于步骤1202中训练NWDAF向推断NWDAF提供的模型的性能评估结果，步骤1204-步骤1206可以不执行。
基于上述实施例，针对推断NWDAF不具备评估功能的场景，当训练NWDAF无法获取推断NWDAF关于模型性能的实时反馈时可以进行周期性再训练，以保证模型的性能。
需要说明的是，基于上述实施例八，在当有多个推断NWDAF时，上述步骤1204中，训练NWDAF可以向多个推断NWDAF发送模型更新请求，从而实现多个推断NWDAF的模型更新。
需要说明的是，基于上述实施例八，当一个分析类型标识对应多个子模型，每个子模型用一个模型标识进行标识，则上述步骤1204中还可以携带一个或多个模型标识，从而实现推断NWDAF中的一个或多个子模型的更新。
实施例九
该实施例九与上述实施例一至实施例八之间的关系是：上述实施例一至实施例八是该实施例九的各种不同的具体实现方式。如图13所示，为本申请实施例提供的一种通信方法流程示意图。需要说明的是，该实施例九中的第一NWDAF可以是上述实施例一至实施例八中的训练NWDAF，第二NWDAF可以是上述实施例一至实施例八中的推断NWDAF1，第三NWDAF可以是上述实施例一至实施例八中的推断NWDAF2。
该方法包括以下步骤：
步骤1301，第一NWDAF向第二NWDAF发送第三信息。相应地，第二NWDAF收到第三信息。
该第三信息包括模型的性能指标，该模型的性能指标用于获取模型的性能的评估结果。可选的，模型性能指标包括以下一项或多项：精确率、准确率、错误率、召回率、F1分数、均方误差、均方根误差、均方根对数误差、平均绝对误差、模型推理时长、模型鲁棒性、模型可扩展性、模型可解释性。也即，第二NWDAF根据接收到的模型的性能指标，对正在使用的模型进行性能评估，进而得到性能的评估结果，并生成模型的性能报告。
可选的，该第三信息还包括以下一项或多项：分析类型标识、模型的标识、子模型的标识。其中，分析类型标识(Analytics ID)用于指示模型的分析类型，比如可以是Service Experience，Network Performance，UE Mobility等。模型的标识用于标识模型。子模型的标识用于标识该模型的子模型。需要说明的是，当该模型没有子模型，则第三信息可以携带模型的标识，且不需要携带子模型的标识，或者第三信息既不携带模型的标识，也不携带子模型的标识。当该模型有子模型，则需要同时携带模型的标识，以及携带一个或多个子模型的标识。需要说明的是，当第三信息携带子模型的标识，则模型的性能指标用于获取该模型的子模型的性能的评估结果。
其中，关于子模型的具体示例，可以参考上述实施例三以及实施例六的描述。
可选的，该第三信息还包括以下一项或多项：上报周期、门限信息。其中，该上报周期用于指示上报模型的性能报告的时间，也即用于指示第二NWDAF基于该上报周期向第一NWDAF上报模型的性能报告。该门限信息用于指示上报模型的性能报告的条件，也即当第二NWDAF得到的模型的评估结果达到了该门限信息对应的门限值，则第二NWDAF向第一NWDAF上报模型的性能报告。
需要说明的是，该步骤1301为可选步骤。当不执行该步骤1301，则可以预先在第二NWDAF上预先配置上述第三信息，或者是由其它网元向第二NWDAF配置上述第三信息。
步骤1302，第二NWDAF向第一NWDAF发送第一信息。相应地，第一NWDAF收到第一信息。
该第一信息包括模型的性能报告，该模型的性能报告用于指示模型的性能的评估结果，或者，该模型的性能报告用于指示模型的性能的评估结果不满足模型的性能指标的要求。
可选的，第一信息还包括模型的性能报告对应的以下一项或多项信息：时间、区域、切片。该时间指的是生成该模型的性能报告的时间范围，该区域指的是该模型的性能报告 对应的区域范围，该切片指的是该模型的性能报告对应的切片信息。
步骤1303，第一NWDAF根据模型的性能报告更新模型的第一模型信息，获得模型的第二模型信息。
步骤1304，第一NWDAF向第二NWDAF发送第二信息。相应地，第二NWDAF收到第二信息。
该第二信息包括第二模型信息。
可选的，第二信息还包括以下一项或多项：模型的标识、子模型的标识、模型的性能评估结果、模型的性能评估结果对应的硬件能力信息、模型的大小、模型的推理时长。其中，模型的性能评估结果对应的硬件能力信息指的是运行该模型所需要的硬件能力要求，如要求具备图形处理器(Graphic Processing Unit，GPU)加速能力，模型的推理时长指的是该模型接收输入到产生输出之间的时延。可选的，每种硬件能力信息对应一种推理时长，硬件能力越强，则推断时长越短。
步骤1305，第二NWDAF根据第二信息，更新模型。
比如，第二NWDAF基于第二信息，使用第二模型信息替换掉第一模型信息，实现模型更新。
可选的，第一NWDAF还可以向除第二NWDAF之外的其它NWDAF(如第三NWDAF)发送第二信息。也即，由第二NWDAF触发第一NWDAF更新模型，得到第二模型信息，但第一NWDAF不仅将第二信息发送给第二NWDAF，还将第二信息发送给第三NWDAF，以实现第三NWDAF对模型进行更新，从而避免第三NWDAF需要单独向第一NWDAF请求模型更新，可以节约信令开销。
其中，关于第一NWDAF向第三NWDAF发送第二信息的具体示例，可以参考上述实施例二以及实施例五的描述。
作为一种实现方法，上述步骤1301具体可以是：第一NWDAF通过NRF向第二NWDAF发送第三信息，上述步骤1302具体可以是：第一NWDAF通过NRF接收来自第二NWDAF的第一信息，上述步骤1304具体可以是：第一NWDAF通过NRF向第二NWDAF发送第二信息。也即，当第一NWDAF与第二NWDAF之间没有接口时，则可以通过NRF作为中间网元，实现第一NWDAF与第二NWDAF之间的交互。
其中，关于通过NRF作为中间网元的具体示例，可以参考上述实施例一至实施例三的描述。
基于上述方案，当第二NWDAF无法完成模型训练时，则第二NWDAF可以向第一NWDAF发送模型的性能报告，从而第一NWDAF可以根据该模型的性能报告对模型进行更新，得到模型的第二模型信息，并将第二模型信息发送给第二NWDAF，使得第二NWDAF可以基于第二模型信息更新模型，从而可以实现在模型性能下降时及时对模型进行训练，进而可以保证模型性能。
示例性的，假设模型为业务体验模型，该模型可以用于基于业务流对应的网络数据(比如业务流对应的终端在基站侧的空口质量、该业务流对应的终端的会话的服务质量流在用户面管理网元上的带宽、时延、抖动等)评估该业务流的业务体验，网络侧策略控制网元(Policy Charging Function，PCF)可以根据该模型的业务体验输出结果确定该业务流的体验要求是否满足，如果不满足，则可以调整该业务的QoS参数。这里，PCF进行QoS参数调整的前提是业务体验模型性能足够好，否则会影响业务体验。比如，以语音业务的业 务体验，也就是MOS分(Mean Opinion Score，平均意见分)为例，MOS分要求为3.0分，如果业务流的实际MOS分为2.5分，但是模型的输出MOS分为3.5分，那么PCF不会对业务的QoS参数进行调整，这样就导致业务体验很差，如果模型的性能足够好，那么模型的输出MOS分应该为2.5分，这样PCF就会对业务QoS参数进行调整，使得MOS分达到3.0分以上。针对该示例，模型的性能影响业务体验。并且如果模型性能持续下降，最终可能会恶化到模型完全不可用的程度，造成极差的业务体验或者业务中断。
联邦学习作为一种新型人工智能技术，可以在原始数据不出本域的情况下实现模型跨域联合训练，既可以提高训练的效率，最重要的，可以通过联邦学习技术，避免数据汇聚到数据分析中心时带来的安全问题(比如，原始数据在传输过程中被劫持，原始数据被数据中心错误使用等)。横向联邦学习，作为一种联邦学习技术，适合“特征重复度非常高，但是数据样本之间差异较大”的训练数据场景。
如图14(a)所示，为横向联邦学习的训练过程(以线性回归为例)。可以看到，横向联邦包括一个中心服务器(server)节点以及多个边缘客户端(client)节点(例如，client节点A、client节点B以及client节点K)，这其中，原始数据都分布在各个client节点，server节点不具有原始数据，并且client节点不允许将原始数据发送给server节点。
首先，各个client节点上的数据集(假设共K个client节点，也就是存在K个数据集)分别是：
Figure PCTCN2020117940-appb-000001
其中，x为样本数据，y为样本数据对应的标签数据。横向联邦学习中每个样本数据都包括标签，即标签和数据存放在一起。
然后，每个client节点上的数据分析模块可以根据线性回归算法各自训练自己的模型，称之为子模型，即：
h(x i)＝Θ Ax i A，h(x j)＝Θ Bx i B，...，h(x K)＝Θ KKx k K。
假设线性回归所使用的损失函数是均方误差(Mean Squared Error，MSE)，那么每个
子模型训练的目标函数(整个训练的过程就是使得上述损失函数的值最小)为：
Figure PCTCN2020117940-appb-000002
下面才真正开始训练过程，针对每一次迭代过程，
(1)每个client节点生成的子模型梯度如下：
Figure PCTCN2020117940-appb-000003
(2)每个client上报样本个数以及本地梯度值，即：
N I以及
Figure PCTCN2020117940-appb-000004
其中，N I表示样本个数，
Figure PCTCN2020117940-appb-000005
表示本地梯度值。
(3)server节点收到上述信息后，对梯度进行聚合，如下：
Figure PCTCN2020117940-appb-000006
其中，||K||为client节点的个数，P I＝N I/∑ IN I。
(4)server节点将聚合后的梯度下发给每一个参与训练的client节点，然后client节点本地更新模型参数，如下：
Figure PCTCN2020117940-appb-000007
(5)client节点进行模型参数更新后，计算损失函数值L I，转至步骤(1)。
上述训练过程，server节点可以通过迭代次数控制训练结束，比如训练10000次终止训练，或者通过设置损失函数的阈值控制训练结束，比如L I≤0.0001时，训练结束。
训练结束后，每个client节点都会保留着同一份模型(可以来自server节点，也可以是本地进一步根据来自server节点本地个性化所得)，用于本地推理。
本申请实施例可以将横向联邦学习与NWDAF相结合，实现模型训练与更新过程。其中，第一NWDAF(也称为Server NWDAF)可以训练模型或者聚合模型，第二NWDAF(也称为Client NWDAF)可以训练模型、更新模型以及使用模型进行推断。
如图14(b)所示，为本申请实施例提供的又一种通信方法流程示意图。该方法包括以下步骤：
步骤1401，第一NWDAF向NRF注册。
第一NWDAF向NRF发送NF注册请求，携带NF配置文件，该NF配置文件包括NF类型，NF服务名称(NF Service,如ModelProvision)，分析类型标识(Analytics ID)等信息。
相应地，NRF存储NF配置文件以及向第一NWDAF发送NF注册响应。
步骤1402，第二NWDAF向NRF注册。
第二NWDAF向NRF发送NF注册请求，携带NF配置文件，该NF配置文件包括NF类型，NF服务名称(NF Service，如ModelUpdate)，分析类型标识(Analytics ID)等信息。
相应地，NRF存储NF配置文件以及向第二NWDAF发送NF注册响应。
步骤1403，第二NWDAF向NRF发送NF发现请求，携带NF配置文件。
比如携带的NF配置文件包含NF类型(如NWDAF)，NF服务名称(NF Service，如ModelProvision，以及携带Analytics ID，则该NF发现请求用于请求从NRF获取对应该Analytics ID的Server NWDAF。
步骤1404，NRF向第二NWDAF发送NF发现响应，其中携带NWDAF实例。
其中，携带的NWDAF实例是Server NWDAF的一个实例，可以用Server NWDAF的ID或IP地址来表示。步骤1405，第一NWDAF向NRF发送NF发现请求，携带NF配置文件。
比如携带的NF配置文件包含NF类型(如NWDAF)，NF服务名称(NF Service，如ModelUpdate)，以及携带Analytics ID，则该NF发现请求用于请求从NRF获取对应该Analytics ID的Client NWDAF。
步骤1406，NRF向第一NWDAF发送NF发现响应，其中携带NWDAF实例。
其中，携带的NWDAF实例是Client NWDAF的一个实例，可以用Client NWDAF的ID或IP地址来表示。
需要说明的是，NF发现响应可以包含一个或多个Client NWDAF实例。
需要说明的是，“步骤1403-步骤1404”以及“步骤1405-步骤1406”可以只执行其中一个，这样，在联邦学习中，可以是Client NWDAF主动触发到Server NWDAF的横向联邦训练，也可以是Server NWDAF主动触发到Client NWDAF横向联邦训练。
其中，上述步骤1401-步骤1406可选步骤。例如，第一NWDAF中配置了第二NWDAF的NF配置文件和/或第二NWDAF中配置了第一NWDAF的NF配置文件，步骤1401-步骤1406可以不执行。
步骤1407，第二NWDAF向第一NWDAF发送模型订阅请求，携带Analytics ID。
该模型订阅请求用于向第一NWDAF订阅该Analytics ID对应的模型索引信息。
步骤1408，第一NWDAF向第二NWDAF发送模型通知1，携带模型索引信息1。
该模型索引信息1即为与Analytics ID对应的模型的索引信息。
该模型通知为与步骤1707的模型订阅请求对应的模型通知。
进一步的，第二NWDAF可以根据该模型索引信息1，获取到相应的模型的第一信息。
步骤1409，第一NWDAF向第二NWDAF发送模型订阅请求，携带模型索引信息1。
该模型订阅请求用于向第二NWDAF请求更新模型索引信息1对应的模型的第一信息，并订阅更新的模型的信息。
步骤1410，模型更新。
具体的，第二NWDAF利用模型索引信息1对应的模型信息进行本地训练，获得的模型的第二信息，并确定的模型的第二信息对应的模型索引信息2。
步骤1411，第二NWDAF向第一NWDAF发送模型通知，携带模型索引信息2。
该模型通知为与步骤1409的模型订阅请求对应的模型通知。
步骤1412，模型更新。
具体的，第一NWDAF利用模型索引信息2对应的模型的第二信息进行本地训练，获得的模型的第三信息，并确定的模型的第三信息对应的模型索引信息3。
可选的，步骤1407-1410中的第二NWDAF可以是多个Client NWDAF的实例，则第一NWDAF在步骤1411可以接收到来自多个第二NWDAF实例的模型索引信息，第一NWDAF根据多个模型索引信息获取对应的多个模型信息，并将多个模型信息进行聚合训练得到更新的模型信息。
步骤1413，第一NWDAF向第二NWDAF发送模型通知，携带模型索引信息3。
该模型通知为与步骤1407的模型订阅请求1对应的模型通知。
后续，可以重复上述步骤1410至步骤1413，并且模型索引信息一直在发生变化，直到第一NWDAF确定停止迭代。可选的，第一NWDAF可以向第二NWDAF发送模型取消订阅消息，也即取消步骤1409对应的模型订阅请求，实现停止迭代。
需要说明的是，该实施例中，模型索引信息可以包括标识信息，该标识信息用于指示该模型索引信息对应的模型的信息。可选的，模型索引信息还包括以下一项或多项：模型对应的分析类型标识、模型的标识、模型的信息的版本信息。
基于上述方案，第一NWDAF与第二NWDAF均可以对模型进行更新得到新的模型信息，并将新的模型信息对应的模型索引信息发送给对方，如此可以实现模型反复迭代，从而可以实现模型性能的逐步提升，最终得到一个模型性能最优的模型，进而可以保证模型性能。
参考图15，为本申请实施例提供的一种通信装置示意图，该通信装置1500包括收发单元1510和处理单元1520。
在第一个实施例中，该通信装置用于实现上述各实施例中对应第一数据分析网元的各 个步骤：
收发单元1510，用于接收来自第二数据分析网元的第一信息，所述第一信息包括模型的性能报告，所述模型的性能报告用于指示所述模型的性能的评估结果，或者，所述模型的性能报告用于指示所述模型的性能的评估结果不满足所述模型的性能指标的要求；以及，用于向所述第二数据分析网元发送第二信息，所述第二信息包括所述模型的第二模型信息。处理单元1520，用于根据所述模型的性能报告更新所述模型的第一模型信息，获得所述第二模型信息。
在一种可能的实现方法中，所述收发单元1510，还用于向所述第二数据分析网元发送第三信息，所述第三信息包括所述模型的性能指标，所述模型的性能指标用于获取所述模型的性能的评估结果。
在一种可能的实现方法中，所述收发单元1510，还用于向第三数据分析网元发送所述第二信息。
在一种可能的实现方法中，所述收发单元1510，用于接收来自第二数据分析网元的第一信息，具体包括：用于通过网络存储网元接收来自所述第二数据分析网元的所述第一信息。所述收发单元1510，用于向所述第二数据分析网元发送第二信息，具体包括：用于通过网络存储网元向所述第二数据分析网元发送所述第二信息。
在一种可能的实现方法中，所述模型性能指标包括以下一项或多项：精确率、准确率、错误率、召回率、F1分数、均方误差、均方根误差、均方根对数误差、平均绝对误差、模型推理时长、模型鲁棒性、模型可扩展性、模型可解释性。
在一种可能的实现方法中，所述第三信息还包括以下一项或多项：分析类型标识、所述模型的标识、子模型的标识，所述分析类型标识用于指示所述模型的分析类型。
在一种可能的实现方法中，所述第三信息还包括以下一项或多项：上报周期、门限信息，所述上报周期用于指示上报所述模型的性能报告的时间，所述门限信息用于指示上报所述模型的性能报告的条件。
在一种可能的实现方法中，所述第一信息还包括所述模型的性能报告对应的以下一项或多项信息：时间、区域、切片。
在一种可能的实现方法中，所述第二信息还包括以下一项或多项：所述模型的标识、子模型的标识、所述模型的性能评估结果、所述模型的性能评估结果对应的硬件能力信息、所述模型的大小、所述模型的推理时长。
在第二个实施例中，该通信装置用于实现上述各实施例中对应第二数据分析网元的各个步骤：
收发单元1510，用于向第一数据分析网元发送第一信息，所述第一信息包括模型的性能报告，所述模型的性能报告用于指示所述模型的性能的评估结果，或者，所述模型的性能报告用于指示所述模型的性能的评估结果不满足所述模型的性能指标的要求；以及，用于接收来自所述第一数据分析网元的第二信息，所述第二信息包括所述模型的第二模型信息，所述模型的第二信息是根据所述模型的性能报告更新所述模型的第一模型信息得到的。处理单元1520，用于根据所述第二模型信息，更新所述模型。
在一种可能的实现方法中，所述收发单元1510，还用于接收来自所述第一数据分析网元的第三信息，所述第三信息包括所述模型的性能指标，所述模型的性能指标用于获取所述模型的性能的评估结果。
在一种可能的实现方法中，所述收发单元1510，用于向第一数据分析网元发送第一信息，具体包括：用于通过网络存储网元向所述第一数据分析网元发送所述第一信息。所述收发单元1510，用于接收来自所述第一数据分析网元的第二信息，具体包括：用于通过网络存储网元接收来自所述第一数据分析网元的第二信息。
在一种可能的实现方法中，所述模型性能指标包括以下一项或多项：精确率、准确率、错误率、召回率、F1分数、均方误差、均方根误差、均方根对数误差、平均绝对误差、模型推理时长、模型鲁棒性、模型可扩展性、模型可解释性。
在一种可能的实现方法中，所述第三信息还包括以下一项或多项：分析类型标识、所述模型的标识、子模型的标识，所述分析类型标识用于指示所述模型的分析类型。
在一种可能的实现方法中，所述第三信息还包括以下一项或多项：上报周期、门限信息，所述上报周期用于指示上报所述模型的性能报告的时间，所述门限信息用于指示上报所述模型的性能报告的条件。
在一种可能的实现方法中，所述第一信息还包括所述模型的性能报告对应的以下一项或多项信息：时间、区域、切片。
在一种可能的实现方法中，所述第二信息还包括以下一项或多项：所述模型的标识、子模型的标识、所述模型的性能评估结果、所述模型的性能评估结果对应的硬件能力信息、所述模型的大小、所述模型的推理时长。
可选地，上述通信装置还可以包括存储单元，该存储单元用于存储数据或者指令(也可以称为代码或者程序)，上述各个单元可以和存储单元交互或者耦合，以实现对应的方法或者功能。例如，处理单元1520可以读取存储单元中的数据或者指令，使得通信装置实现上述实施例中的方法。
应理解以上通信装置中单元的划分仅仅是一种逻辑功能的划分，实际实现时可以全部或部分集成到一个物理实体上，也可以物理上分开。且通信装置中的单元可以全部以软件通过处理元件调用的形式实现；也可以全部以硬件的形式实现；还可以部分单元以软件通过处理元件调用的形式实现，部分单元以硬件的形式实现。例如，各个单元可以为单独设立的处理元件，也可以集成在通信装置的某一个芯片中实现，此外，也可以以程序的形式存储于存储器中，由通信装置的某一个处理元件调用并执行该单元的功能。此外这些单元全部或部分可以集成在一起，也可以独立实现。这里所述的处理元件又可以成为处理器，可以是一种具有信号的处理能力的集成电路。在实现过程中，上述方法的各步骤或以上各个单元可以通过处理器元件中的硬件的集成逻辑电路实现或者以软件通过处理元件调用的形式实现。
在一个例子中，以上任一通信装置中的单元可以是被配置成实施以上方法的一个或多个集成电路，例如：一个或多个特定集成电路(application specific integrated circuit，ASIC)，或，一个或多个微处理器(digital singnal processor，DSP)，或，一个或者多个现场可编程门阵列(field programmable gate array，FPGA)，或这些集成电路形式中至少两种的组合。再如，当通信装置中的单元可以通过处理元件调度程序的形式实现时，该处理元件可以是通用处理器，例如中央处理器(central processing unit，CPU)或其它可以调用程序的处理器。再如，这些单元可以集成在一起，以片上系统(system-on-a-chip，SOC)的形式实现。
参考图16，为本申请实施例提供的一种通信装置示意图，用于实现以上实施例中第 一数据分析网元或第二数据分析网元的操作。如图16所示，该通信装置包括：处理器1610和接口1630，可选地，该通信装置还包括存储器1620。接口1630用于实现与其他设备进行通信。
以上实施例中第一数据分析网元或第二数据分析网元执行的方法可以通过处理器1610调用存储器(可以是第一数据分析网元或第二数据分析网元中的存储器1620，也可以是外部存储器)中存储的程序来实现。即，第一数据分析网元或第二数据分析网元可以包括处理器1610，该处理器1610通过调用存储器中的程序，以执行以上方法实施例中第一数据分析网元或第二数据分析网元执行的方法。这里的处理器可以是一种具有信号的处理能力的集成电路，例如CPU。第一数据分析网元或第二数据分析网元可以通过配置成实施以上方法的一个或多个集成电路来实现。例如：一个或多个ASIC，或，一个或多个微处理器DSP，或，一个或者多个FPGA等，或这些集成电路形式中至少两种的组合。或者，可以结合以上实现方式。
具体的，图15中的收发单元1510和处理单元1520的功能/实现过程可以通过图16所示的通信装置1600中的处理器1610调用存储器1620中存储的计算机可执行指令来实现。或者，图15中的处理单元1520的功能/实现过程可以通过图16所示的通信装置1600中的处理器1610调用存储器1620中存储的计算机执行指令来实现，图15中的收发单元1510的功能/实现过程可以通过图16中所示的通信装置1600中的接口1630来实现，示例性的，收发单元1510的功能/实现过程可以通过处理器调用存储器中的程序指令以驱动接口1630来实现。
本领域普通技术人员可以理解：本申请中涉及的第一、第二等各种数字编号仅为描述方便进行的区分，并不用来限制本申请实施例的范围，也表示先后顺序。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。字符“/”一般表示前后关联对象是一种“或”的关系。“至少一个”是指一个或者多个。至少两个是指两个或者多个。“至少一个”、“任意一个”或其类似表达，是指的这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a,b,或c中的至少一项(个、种)，可以表示：a,b,c,a-b,a-c,b-c,或a-b-c，其中a,b,c可以是单个，也可以是多个。“多个”是指两个或两个以上，其它量词与之类似。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本发明实施例的实施过程构成任何限定。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、 数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包括一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘(solid state disk，SSD))等。
本申请实施例中所描述的各种说明性的逻辑单元和电路可以通过通用处理器，数字信号处理器，专用集成电路(ASIC)，现场可编程门阵列(FPGA)或其它可编程逻辑装置，离散门或晶体管逻辑，离散硬件部件，或上述任何组合的设计来实现或操作所描述的功能。通用处理器可以为微处理器，可选地，该通用处理器也可以为任何传统的处理器、控制器、微控制器或状态机。处理器也可以通过计算装置的组合来实现，例如数字信号处理器和微处理器，多个微处理器，一个或多个微处理器联合一个数字信号处理器核，或任何其它类似的配置来实现。
本申请实施例中所描述的方法或算法的步骤可以直接嵌入硬件、处理器执行的软件单元、或者这两者的结合。软件单元可以存储于随机存取存储器(Random Access Memory，RAM)、闪存、只读存储器(Read-Only Memory，ROM)、EPROM存储器、EEPROM存储器、寄存器、硬盘、可移动磁盘、CD-ROM或本领域中其它任意形式的存储媒介中。示例性地，存储媒介可以与处理器连接，以使得处理器可以从存储媒介中读取信息，并可以向存储媒介存写信息。可选地，存储媒介还可以集成到处理器中。处理器和存储媒介可以设置于ASIC中。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
在一个或多个示例性的设计中，本申请所描述的上述功能可以在硬件、软件、固件或这三者的任意组合来实现。如果在软件中实现，这些功能可以存储与电脑可读的媒介上，或以一个或多个指令或代码形式传输于电脑可读的媒介上。电脑可读媒介包括电脑存储媒介和便于使得让电脑程序从一个地方转移到其它地方的通信媒介。存储媒介可以是任何通用或特殊电脑可以接入访问的可用媒体。例如，这样的电脑可读媒体可以包括但不限于RAM、ROM、EEPROM、CD-ROM或其它光盘存储、磁盘存储或其它磁性存储装置，或其它任何可以用于承载或存储以指令或数据结构和其它可被通用或特殊电脑、或通用或特殊处理器读取形式的程序代码的媒介。此外，任何连接都可以被适当地定义为电脑可读媒介，例如，如果软件是从一个网站站点、服务器或其它远程资源通过一个同轴电缆、光纤电脑、双绞线、数字用户线(DSL)或以例如红外、无线和微波等无线方式传输的也被包含在所定义的电脑可读媒介中。所述的碟片(disk)和磁盘(disc)包括压缩磁盘、镭射盘、光盘、数字通用光盘(英文：Digital Versatile Disc，简称：DVD)、软盘和蓝光光盘，磁盘通常以磁性复制数据，而碟片通常以激光进行光学复制数据。上述的组合也可以包含在电脑可读媒介中。
尽管结合具体特征及其实施例对本申请进行了描述，显而易见的，在不脱离本申请的精神和范围的情况下，可对其进行各种修改和组合。相应地，本说明书和附图仅仅是所附权利要求所界定的本申请的示例性说明，且视为已覆盖本申请范围内的任意和所有修改、变化、组合或等同物。显然，本领域的技术人员可以对本申请进行各种改动和变型而不脱 离本申请的范围。这样，倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内，则本申请也意图包括这些改动和变型在内。

Claims (22)

  1. 一种通信方法，其特征在于，包括：
    第一数据分析网元接收来自第二数据分析网元的第一信息，所述第一信息包括模型的性能报告，所述模型的性能报告用于指示所述模型的性能的评估结果，或者，所述模型的性能报告用于指示所述模型的性能的评估结果不满足所述模型的性能指标的要求；
    所述第一数据分析网元根据所述模型的性能报告更新所述模型的第一模型信息，获得所述模型的第二模型信息；
    所述第一数据分析网元向所述第二数据分析网元发送第二信息，所述第二信息包括所述第二模型信息。
  2. 如权利要求1所述的方法，其特征在于，还包括：
    所述第一数据分析网元向所述第二数据分析网元发送第三信息，所述第三信息包括所述模型的性能指标，所述模型的性能指标用于获取所述模型的性能的评估结果。
  3. 如权利要求2所述的方法，其特征在于，所述模型性能指标包括以下一项或多项：精确率、准确率、错误率、召回率、F1分数、均方误差、均方根误差、均方根对数误差、平均绝对误差、模型推理时长、模型鲁棒性、模型可扩展性、模型可解释性。
  4. 如权利要求2或3所述的方法，其特征在于，所述第三信息还包括以下一项或多项：分析类型标识、所述模型的标识、子模型的标识，所述分析类型标识用于指示所述模型的分析类型。
  5. 如权利要求2-4任一所述的方法，其特征在于，所述第三信息还包括以下一项或多项：上报周期、门限信息，所述上报周期用于指示上报所述模型的性能报告的时间，所述门限信息用于指示上报所述模型的性能报告的条件。
  6. 如权利要求1-5任一所述的方法，其特征在于，所述第一信息还包括所述模型的性能报告对应的以下一项或多项信息：时间、区域、切片。
  7. 如权利要求1-6任一所述的方法，其特征在于，所述第二信息还包括以下一项或多项：所述模型的标识、子模型的标识、所述模型的性能评估结果、所述模型的性能评估结果对应的硬件能力信息、所述模型的大小、所述模型的推理时长。
  8. 如权利要求1-7任一所述的方法，其特征在于，还包括：
    所述第一数据分析网元向第三数据分析网元发送所述第二信息。
  9. 如权利要求1-8任一所述的方法，其特征在于，所述第一数据分析网元接收来自第二数据分析网元的第一信息，包括：
    所述第一数据分析网元通过网络存储网元接收来自所述第二数据分析网元的所述第一信息；
    所述第一数据分析网元向所述第二数据分析网元发送第二信息，包括：
    所述第一数据分析网元通过网络存储网元向所述第二数据分析网元发送所述第二信息。
  10. 一种通信方法，其特征在于，包括：
    第二数据分析网元向第一数据分析网元发送第一信息，所述第一信息包括模型的性能报告，所述模型的性能报告用于指示所述模型的性能的评估结果，或者，所述模型的性能报告用于指示所述模型的性能的评估结果不满足所述模型的性能指标的要求；
    所述第二数据分析网元接收来自所述第一数据分析网元的第二信息，所述第二信息包括所述模型的第二模型信息，所述模型的第二信息是根据所述模型的性能报告更新所述模型的第一模型信息得到的；
    所述第二数据分析网元根据所述第二模型信息，更新所述模型。
  11. 如权利要求10所述的方法，其特征在于，还包括：
    所述第二数据分析网元接收来自所述第一数据分析网元的第三信息，所述第三信息包括所述模型的性能指标，所述模型的性能指标用于获取所述模型的性能的评估结果。
  12. 如权利要求11所述的方法，其特征在于，所述模型性能指标包括以下一项或多项：精确率、准确率、错误率、召回率、F1分数、均方误差、均方根误差、均方根对数误差、平均绝对误差、模型推理时长、模型鲁棒性、模型可扩展性、模型可解释性。
  13. 如权利要求11或12所述的方法，其特征在于，所述第三信息还包括以下一项或多项：分析类型标识、所述模型的标识、子模型的标识，所述分析类型标识用于指示所述模型的分析类型。
  14. 如权利要求11-13任一所述的方法，其特征在于，所述第三信息还包括以下一项或多项：上报周期、门限信息，所述上报周期用于指示上报所述模型的性能报告的时间，所述门限信息用于指示上报所述模型的性能报告的条件。
  15. 如权利要求10-14任一所述的方法，其特征在于，所述第一信息还包括所述模型的性能报告对应的以下一项或多项信息：时间、区域、切片。
  16. 如权利要求10-15任一所述的方法，其特征在于，所述第二信息还包括以下一项或多项：所述模型的标识、子模型的标识、所述模型的性能评估结果、所述模型的性能评估结果对应的硬件能力信息、所述模型的大小、所述模型的推理时长。
  17. 如权利要求10-16任一所述的方法，其特征在于，所述第二数据分析网元向第一数据分析网元发送第一信息，包括：
    所述第二数据分析网元通过网络存储网元向所述第一数据分析网元发送所述第一信息；
    所述第二数据分析网元接收来自所述第一数据分析网元的第二信息，包括：
    所述第二数据分析网元通过网络存储网元接收来自所述第一数据分析网元的第二信息。
  18. 一种通信装置，其特征在于，包括：处理器，所述处理器与存储器耦合，所述存储器用于存储程序或指令，当所述程序或指令被所述处理器执行时，使得所述装置执行如权利要求1-9任一所述的方法。
  19. 一种通信装置，其特征在于，包括：处理器，所述处理器与存储器耦合，所述存储器用于存储程序或指令，当所述程序或指令被所述处理器执行时，使得所述装置执行如权利要求10-17任一所述的方法。
  20. 一种芯片系统，其特征在于，包括：所述芯片系统包括至少一个处理器，和接口电路，所述接口电路和所述至少一个处理器耦合，所述处理器通过运行指令，以执行权利要求1-9任一所述的方法。
  21. 一种芯片系统，其特征在于，包括：所述芯片系统包括至少一个处理器，和接口电路，所述接口电路和所述至少一个处理器耦合，所述处理器通过运行指令，以执行权利要求10-17任一所述的方法。
  22. 一种通信系统，其特征在于，包括用于执行权利要求1-9任一所述方法的第一数据分析网元，和用于执行权利要求10-17任一所述方法的第二数据分析网元。
PCT/CN2020/117940 2020-09-25 2020-09-25 通信方法、装置及系统 WO2022061784A1 (zh)

Priority Applications (7)

Application Number Priority Date Filing Date Title
PCT/CN2020/117940 WO2022061784A1 (zh) 2020-09-25 2020-09-25 通信方法、装置及系统
CA3193840A CA3193840A1 (en) 2020-09-25 2021-04-02 Communication method, apparatus and system
AU2021347699A AU2021347699A1 (en) 2020-09-25 2021-04-02 Communication method, apparatus and system
PCT/CN2021/085428 WO2022062362A1 (zh) 2020-09-25 2021-04-02 通信方法、装置及系统
CN202180041882.4A CN115699848A (zh) 2020-09-25 2021-04-02 通信方法、装置及系统
EP21870751.1A EP4207860A4 (en) 2020-09-25 2021-04-02 COMMUNICATION METHOD, APPARATUS AND SYSTEM
US18/188,205 US20230224752A1 (en) 2020-09-25 2023-03-22 Communication method, apparatus, and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/117940 WO2022061784A1 (zh) 2020-09-25 2020-09-25 通信方法、装置及系统

Publications (1)

Publication Number Publication Date
WO2022061784A1 true WO2022061784A1 (zh) 2022-03-31

Family

ID=80844486

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/CN2020/117940 WO2022061784A1 (zh) 2020-09-25 2020-09-25 通信方法、装置及系统
PCT/CN2021/085428 WO2022062362A1 (zh) 2020-09-25 2021-04-02 通信方法、装置及系统

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/085428 WO2022062362A1 (zh) 2020-09-25 2021-04-02 通信方法、装置及系统

Country Status (6)

Country Link
US (1) US20230224752A1 (zh)
EP (1) EP4207860A4 (zh)
CN (1) CN115699848A (zh)
AU (1) AU2021347699A1 (zh)
CA (1) CA3193840A1 (zh)
WO (2) WO2022061784A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023213286A1 (zh) * 2022-05-05 2023-11-09 维沃移动通信有限公司 模型标识管理方法、装置及存储介质

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20220001797A (ko) * 2020-06-30 2022-01-06 삼성전자주식회사 무선 통신 네트워크에서 네트워크 분석 정보 제공 방법 및 장치
WO2023213288A1 (zh) * 2022-05-05 2023-11-09 维沃移动通信有限公司 模型获取方法及通信设备
CN117082564A (zh) * 2022-05-06 2023-11-17 华为技术有限公司 一种通信方法和装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110312279A (zh) * 2018-03-27 2019-10-08 电信科学技术研究院有限公司 一种网络数据的监测方法及装置
US20190394655A1 (en) * 2018-06-22 2019-12-26 Huawei Technologies Co., Ltd. Data analytics management (dam), configuration specification and procedures, provisioning, and service based architecture (sba)
CN110831029A (zh) * 2018-08-13 2020-02-21 华为技术有限公司 一种模型的优化方法和分析网元
US20200244557A1 (en) * 2017-10-23 2020-07-30 Huawei Technologies Co., Ltd. Traffic processing method, user plane apparatus, and terminal device

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190294320A1 (en) * 2018-06-16 2019-09-26 Moshe Guttmann Class aware object marking tool
US10750371B2 (en) * 2018-12-12 2020-08-18 Verizon Patent And Licensing, Inc. Utilizing machine learning to provide closed-loop network management of a fifth generation (5G) network

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200244557A1 (en) * 2017-10-23 2020-07-30 Huawei Technologies Co., Ltd. Traffic processing method, user plane apparatus, and terminal device
CN110312279A (zh) * 2018-03-27 2019-10-08 电信科学技术研究院有限公司 一种网络数据的监测方法及装置
US20190394655A1 (en) * 2018-06-22 2019-12-26 Huawei Technologies Co., Ltd. Data analytics management (dam), configuration specification and procedures, provisioning, and service based architecture (sba)
CN110831029A (zh) * 2018-08-13 2020-02-21 华为技术有限公司 一种模型的优化方法和分析网元

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
CHINA MOBILE: "Solution: Predictable network performance", 3GPP DRAFT; S2-188030 SOLUTION PREDICTABLE NETWORK PERFORMANCE, 3RD GENERATION PARTNERSHIP PROJECT (3GPP), MOBILE COMPETENCE CENTRE ; 650, ROUTE DES LUCIOLES ; F-06921 SOPHIA-ANTIPOLIS CEDEX ; FRANCE, vol. SA WG2, no. Sophia Antipolis, France; 20180820 - 20180824, 14 August 2018 (2018-08-14), Mobile Competence Centre ; 650, route des Lucioles ; F-06921 Sophia-Antipolis Cedex ; France , XP051536976 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023213286A1 (zh) * 2022-05-05 2023-11-09 维沃移动通信有限公司 模型标识管理方法、装置及存储介质

Also Published As

Publication number Publication date
EP4207860A1 (en) 2023-07-05
CA3193840A1 (en) 2022-03-31
EP4207860A4 (en) 2024-02-28
US20230224752A1 (en) 2023-07-13
CN115699848A (zh) 2023-02-03
WO2022062362A1 (zh) 2022-03-31
AU2021347699A1 (en) 2023-05-18

Similar Documents

Publication Publication Date Title
WO2022061784A1 (zh) 通信方法、装置及系统
EP4132066A1 (en) Communication method, apparatus and system
US11418413B2 (en) Sharable storage method and system for network data analytics
CA3112926C (en) Slice information processing method and apparatus
US11265205B2 (en) System and method for anomaly detection with root cause identification
WO2021155579A1 (zh) 一种数据分析方法、装置及系统
WO2022033115A1 (zh) 一种通信方法和通信装置
WO2020108002A1 (zh) 一种传输策略确定方法、策略控制方法及装置
WO2019206100A1 (zh) 一种特征工程编排方法及装置
WO2019057027A1 (zh) 大数据分析服务提供方法和装置、计算机可读存储介质
CN116325686A (zh) 一种通信方法和装置
WO2022188670A1 (zh) 网络分析转移方法、装置及网络功能实体
US20240064211A1 (en) Method for Notification of Network Function Producer (NFp) Quality
WO2023141834A1 (zh) 模型性能的监测方法、装置、设备及介质
WO2023185711A1 (zh) 一种用于机器学习模型训练的通信方法及装置
EP4087192A1 (en) Communication network arrangement and method for providing a machine learning model for performing communication network analytics
WO2024032270A1 (zh) 一种策略确定方法、装置及存储介质
WO2022141279A1 (zh) 业务体验模型的确定方法及通信装置
WO2023098341A1 (zh) 异常信令的管控方法、装置、电子设备和存储介质
WO2024032031A1 (zh) 一种数据分析方法及装置
WO2024099016A1 (zh) 一种通信方法及装置
WO2024027427A1 (zh) 异常检测的方法和通信装置
EP4135413A1 (en) Communication network arrangement and method for selecting a network component
KR20230113191A (ko) 이동 통신 시스템의 훈련 단말 선택 장치 및 방법
CN116938747A (zh) 一种通信方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20954636

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20954636

Country of ref document: EP

Kind code of ref document: A1